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This  research  uses  detailed  literature  reviews  and  expert  interviews  to  identify  the 
objectives  and  performance  measurement  procedures  of  Plebe  Summer.  Next  it  evaluates  the 
performance  measurement  and  outcome  management  procedures  employed  by  the  Naval 
Academy  in  assessing  Plebe  Summer  success.  Evaluated  in  this  research  are  the  objectives, 
outcomes,  outcome  indicators,  data  collection  procedures,  and  benchmarks  applied  to  the  Plebe 
Summer  program.  The  research  indicates  that  a  performance  measurement  and  outcome 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  United  States  Naval  Academy  (USNA)  is  the  premier  source  of  officers  for 
the  Naval  Service.  It  provides  a  four-year  total- immersion  program  where  a  strong, 
balanced  academic  program,  focusing  on  the  education  needs  of  the  Naval  Service  is 
superimposed  on  a  strict,  professional  military  training  environment  (USNA,  1997).  It  is 
the  mission  of  the  Naval  Academy  to: 

Develop  midshipmen  morally,  mentally,  and  physically  and  to  imbue  them 
with  the  highest  ideals  of  duty,  honor  and  loyalty  in  order  to  provide 
graduates  who  are  dedicated  to  a  career  of  naval  service  and  have  potential 
for  future  development  in  mind  and  character  to  assume  the  highest 
responsibilities  of  command,  citizenship,  and  government.  (USNA,  2001c, 
p.  3) 

Throughout  the  more  than  150- year  history  of  the  Naval  Academy  the  institution 
has  evolved  from  a  trade  school  focused  on  the  arts  of  seamanship  to  a  highly  regarded 
learning  institution.  The  Naval  Academy  today  provides  academic  majors  in  19  areas 
including  Engineering,  Sciences,  Humanities,  and  Economics. 

Fourth  Class  midshipmen  (first-year  freshmen)  are  introduced  to  the  Naval 
Academy  culture  during  an  intense  seven-week  course  known  as  Fourth  Class 
Indoctrination.  This  course  is  more  commonly  referred  to  as  Plebe  Summer  and 
historically  runs  each  year  from  late  June  through  late  August.  The  course  is  designed  to 
be  progressive  and  demanding  training  in  subjects  such  as  military  drill,  physical  fitness, 
weapons,  and  naval  history.  These  topics  have  been  identified  as  those  that  are  required 
to  produce  midshipmen  who  function  effectively  within  the  Academy  and  the  Brigade  of 
Midshipmen  (USNA,  2001b,  p.6). 

Plebe  Summer  is  designed  as  a  three  phase  course  of  instruction: 

1.  First  is  the  Processing  Phase.  This  administrative  phase  is  the  shortest 
phase  and  is  focused  on  insuring  all  midshipmen  quickly  make  the  administrative 
transition  into  the  military  culture.  Areas  of  concern  in  this  phase  include  official 
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welcomes,  berthing  assignment,  uniform  and  equipment  issue,  medical  evaluations,  and 
academic  testing  (USNA,  2001b,  p.8). 

2.  Second  is  the  Forming  Phase.  This  phase  is  slightly  more  arduous 
mentally  and  physically.  This  phase  is  designed  to  instill  a  sense  of  teamwork  and  naval 
heritage  in  the  midshipmen.  Topics  taught  during  this  phase  include,  basic  military 
indoctrination,  close  order  drill,  Navy  rights  and  responsibilities.  Activities  emphasize 
discipline  and  teamwork  (USNA,  2001b,  p.8). 

3.  The  final  phase  is  the  Training  Phase.  This  final  phase  is  the  longest  and 
most  challenging.  The  primary  objective  of  this  phase  is  progressive  development 
through  demanding  training  and  follow-up  to  insure  the  incoming  Plebe  class  meets  the 
personal  and  professional  standards  expected  of  midshipmen  at  the  Naval  Academy 
(USNA,  2001b,  p.  8).  Major  topics  covered  during  this  phase  include  moral  and 
character  development,  code  of  conduct  sailing  and  professional  development,  and 
military  drill. 

Plebe  Summer  provides  a  basic  foundation  for  further  midshipman  growth.  The 
fundamental  purpose  of  Plebe  Summer  is  preparing  midshipmen  for  the  demands  of  the 
four- year  Naval  Academy  challenge;  it  is  a  simply  a  first  step.  Upon  completion  of  Plebe 
Summer  midshipmen  begin  a  rigorous  year  of  training  and  education  (called  Fourth  Class 
Development  System)  designed  to  challenge  them  in  every  aspect  of  their  lives.  The  year 
is  designed  on  the  philosophy  that,  “to  lead,  one  must  first  learn  to  follow.”  (USNA 
200 le,  p.  2) 

Performance  measurement;  and  its  extended  application,  outcome  management, 
are  tools  gaining  increasing  popularity  within  both  the  private  and  public  sectors.  The 
core  principle  of  outcome  management  is  that  program  control  is  made  more  effective 
using  detailed  standards  of  performance  that  are  measured  and  reported.  “Increasingly 
companies  are  moving  from  management  by  opinion  to  management  by  fact  -  that  is, 
away  from  soft  science  approach  to  performance  measurement”  (Harbour,  1997,  p.  8). 
The  ability  of  a  high  quality  performance  measurement  program  will  assist  program 
managers  by  providing  relevant  information  quickly  in  order  to  support  decision.  “The 
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goal  of  any  performance  measurement  system  is  to  provide  the  right  people  with  the  right 
performance-related  information  at  the  right  time”  (Harbour,  1997,  p.  8). 

Persons  accountable  for  programs  apply  performance  measurement  techniques  for 
a  number  of  reasons.  Zigon  (1998)  lists  three  reasons  for  performance  measurement  and 
performance  management  that  apply  to  the  USNA: 

•  You  cannot  manage  what  you  cannot  measure. 

•  You  cannot  improve  what  you  cannot  measure. 

•  High  performance  teams  and  individuals  require  clear  goals. 

•  Reward  systems  require  metrics. 

There  is  clear  evidence  that  performance  measurement  and  outcome  management 
are  important  aspects  of  managing  the  Plebe  Summer  program  at  the  United  States  Naval 
Academy.  Most  fundamental  is  performance  measurement’s  ability  to  assist  in 
evaluating  the  success  of  the  Plebe  Summer  program  in  meeting  its  objectives. 

Performance  metrics  are  usually  linked  with  a  program’s  strategic  plan.  Strategic 
plans  identify  broad  goals  and  objectives  while  performance  metrics  provide  specific 
goals  and  measurable  criteria  that  identify  whether  goals  are  met.  Thus,  performance 
objectives  and  metrics  are  intended  to  provide  focus  for  those  in  management  positions. 
“Leading  organizations  do  not  stop  at  the  gathering  and  analysis  of  performance  data; 
rather,  these  organizations  use  performance  measurement  to  drive  improvements  and 
successfully  translate  strategy  into  action.  In  other  words,  they  use  performance 
measurement  for  managing  their  organizations”  (PEA,  1998,  p.  vii). 

The  Naval  Academy  developed  its  own  detailed  strategic  plan  in  1997.  The  plan 
is  consistently  updated  (in  2001  and  again  in  2002)  in  order  to  maintain  its  relevancy. 
The  plan  was  developed  in  response  to  governmental  direction  to  implement  mandatory 
performance  measurement  systems.  All  government  agencies  were  directed  by  the 
Government  Performance  and  Results  Act  of  1993  (GPRA)  to  develop  comprehensive 
plans  detailing  the  mission,  goals,  and  objectives  of  the  agency.  Moreover,  the  GPRA 
requires  a  description  of  how  these  objectives  are  to  be  met  and  how  performance  goals 
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relate  to  the  objectives  of  the  agency.  (United  States  Congress  1993)  The  USNA  plan’s 
foundation  lies  in  eight  “focus  areas”:  leadership  and  professional  excellence,  physical 
fitness,  naval  heritage,  academy  quality  of  life,  character  building,  effective 
communications,  admissions  excellence  and  academic  excellence  (USNA,  2001c,  p.8). 

Plebe  Summer  program  serves  as  an  indoctrination,  and  should  be  designed  as  an 
integral  part  of  this  strategic  plan.  The  performance  metrics  should  align  with  and 
support  the  goals  of  the  plan.  These  goals  will  be  addressed  later  in  this  thesis. 

B.  PURPOSE 

The  level  at  which  the  Naval  Academy  succeeds  in  meeting  the  Plebe  Summer 
objectives  is  unknown.  The  objectives  of  Plebe  Summer  are  diverse.  In  order  to  assure 
Fourth  Class  midshipmen  succeed  at  the  Naval  Academy,  they  must  be  inculcated  with 
Naval  Academy  and  military  cultures  and  be  provided  the  tools  and  skills  necessary  to 
succeed  within  the  culture  during  the  seven-week  Plebe  Summer  program.  Furthermore, 
they  must  be  prepared  to  function  in  demanding  fields  such  as  watch  standing,  teamwork, 
and  military  drill.  Most  importantly,  upon  completion  of  Plebe  Summer,  the  midshipmen 
must  be  prepared  to  meet  the  challenges  concerning  their  moral,  mental,  and  physical 
development  that  will  continue  as  the  plebes  join  the  brigade  of  midshipmen. 

The  purpose  of  this  study  is  to  evaluate  performance  measurement  and  outcome 
management  systems  of  the  Naval  Academy’s  Pfebe  Summer  Program  and  answer  the 
following  research  questions: 

1.  Primary: 

How  effective  are  the  performance  measurement  procedures  used  in  the 

Plebe  Summer  program  in  evaluating  the  successful  accomplishment  of  Plebe 

Summer  objectives? 

2.  Secondary: 

•  What  are  the  measurable  objectives  of  Plebe  Summer? 

•  How  is  the  success  of  Plebe  Summer  currently  measured? 
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•  What  current  performance  measurement  procedures  are  used  to  assess 
the  success  of  Plebe  Summer  objectives? 

•  How  will  effective  performance  measurement  techniques  increase  the 
effectiveness  of  the  Plebe  Summer  program? 

Deeper  understanding  of  the  performance  measurement  and  outcome  management 
of  Plebe  Summer  program  can  be  beneficial  to  program  managers.  Answering  these 
research  questions  will  provide  the  following  benefits  to  the  Naval  Academy  and  the 
Plebe  Summer  program: 

•  Improving  Plebe  Summer  effectiveness  by  identifying  and  separating 
procedures  that  enhance  outcome  management  from  procedures  that  do  not. 

•  Identifying  objectives  and  outcomes  of  the  Plebe  Summer  Program  that  are 
currently  not  assessed. 

•  Improvements  made  in  the  basic  foundation  of  midshipmen  development  will 
ultimately  lead  to  higher  levels  of  achievement  of  the  Naval  Academy’s 
overall  objectives  and  mission. 

•  Careful  evaluation  of  the  performance  metrics  applied  to  Plebe  Summer  may 
also  illuminate  areas  of  concern  associated  with  incorrect  applications  of 
performance  management.  Additional  motivation  to  close  examination  of 
Plebe  Summer  performance  metrics  is  given  by  Perrin  (1998)  where  he  says 
close  examination  may  identify  flaws  in  performance  management  such  as: 
varying  interpretation  of  terms  and  concepts,  meaningless  measures,  goal 
displacement  where  metrics  become  goals,  lost  focus  on  outcome,  and 
misleading  aggregate  indicators  (p.  5-10). 

C.  METHODOLOGY 

This  research  is  a  qualitative  assessment  of  the  performance  and  outcome 
measurement  processes  used  to  assess  the  USNA’s  Plebe  Summer  program  in  meeting  its 
objectives.  It  includes  literature  reviews  and  personal  semi- structured  interviews  of 
Naval  Academy  Plebe  Summer  experts.  The  research  provides  a  qualitative  assessment 
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of  outcome  measurement  methods  using  Hatry  and  Kopczynski’s  Guide  to  Program 
Outcome  Measurement  (1997)  as  a  standard.  The  process  used  in  this  research  includes 
the  following  steps: 

•  A  thorough  literature  review  of  previous  performance  measurement  research. 
This  review  will  include  books,  journal  articles,  internet,  CD-ROM,  and  other 
electronic  media  materials  dedicated  to  performance  measurement  and 
evaluation. 

•  A  thorough  review  of  all  current  USNA  instructions  and  directives  governing 
the  management  of  the  Plebe  Summer  program.  This  review  will  focus  on  the 
purpose,  objectives,  performance  measurement  and  reporting,  and 
accomplishment  of  Plebe  Summer. 

•  Thorough  semi- structured  interviews  of  expert  Naval  Academy  personnel 
accountable  for  Plebe  Summer  planning,  management,  and  evaluation.  The 
purpose  of  these  interviews  is  to  establish  and  validate  the  performance 
metrics  used  in  the  Plebe  Summer  program. 

•  Evaluation  of  current  USNA  Plebe  Summer  performance  metrics  and  outcome 
measurement  processes  using  Hatry  and  Kopczynski’s  Guide  to  Program 
Outcome  Measurement  (1997)  as  a  standard  of  program  evaluation. 

D.  ORGANIZATION  OF  S  TUD Y 

This  study  is  organized  in  five  chapters.  The  following  briefly  describes  the 
contents  each  chapter  of  this  thesis. 

Chapter  I  introduces  the  four  basic  aspects  of  this  research:  background,  purpose, 
methodology,  and  organization  of  study.  The  background  section  describes  the  function 
of  the  United  States  Naval  Academy  and  the  Naval  Academy’s  mission,  performance 
measurement  as  a  method  of  management  within  both  private  and  public  sectors,  and 
reviews  the  Naval  Academy’s  strategic  plan  focusing  on  its  origins  and  purpose. 

Additionally,  Chapter  I  describes  the  purpose  of  this  research.  It  introduces  the 
Plebe  Summer  program,  as  well  as  identifies  the  major  research  questions  of  this  thesis. 
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Lastly,  this  section  proposes  the  benefits  of  successfully  answering  the  research 
questions. 

Chapter  II  is  divided  into  three  areas.  First,  chapter  two  reviews  previous 
performance  measurement  research  and  its  application  in  the  public  sector.  The  review 
will  focus  on  the  purpose  and  objectives  of  performance  measurement,  terminology, 
current  best  practices,  the  misuse  of  performance  measurement  techniques. 

The  literature  review  continues  with  a  review  of  the  current  USNA  strategic  plan. 
The  focus  of  this  section  is  on  the  plan’s  purpose  and  objectives.  Lastly,  chapter  two 
reviews  the  current  USNA  Hebe  Summer  program.  The  examination  focuses  on  the 
purpose,  objectives,  and  methodology  of  the  Plebe  Summer  program. 

Chapter  III  describes  in  detail  Hatry  and  Kopczyski’s  (1997)  Guide  to  Program 
Outcome  Measurement  for  the  Department  of  Education  This  guide  provides  the  model 
later  used  to  analyze  the  Plebe  Summer  outcome  and  performance  measurement 
processes. 

Chapter  IV  describes  the  qualitative  research  process  of  this  thesis.  This  chapter 
summarizes  the  methodological  philosophy  used  to  develop  the  research  methods.  Also, 
this  chapter  outlines  the  scope  and  boundaries  of  the  research,  explains  the  interview 
selection  process,  and  details  the  questions  used  in  the  semi- structured  interview  process. 

In  chapter  V  Plebe  Summer  performance  measurement  and  outcome  management 
procedures  identified  in  Chapter  II  and  in  expert  interviews  are  evaluated  using  Hatry  and 
Kopczyski’s  (1997)  Guide  to  Program  Outcome  Measurement.  The  results  of  this 
evaluation  are  in  this  chapter.  Additionally,  areas  that  are  not  currently  covered  by 
performance  metrics  are  reported  in  this  chapter. 

Chapter  VI  begins  with  a  summary  of  the  previous  chapters.  This  last  chapter 
continues  with  conclusions  by  making  a  qualitative  decision  answering  the  primary 
research  question.  Recommendations  concerning  Plebe  Summer  performance 
measurement  as  well  as  recommendations  for  further  studies  will  follow  the  conclusions. 
Chapter  VI  will  identify  the  connection  between  performance  measurement,  the  strategic 
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plan,  and  the  management  of  the  Plebe  Summer  program  in  an  effort  to  better  support  the 
thesis’  answers  of  the  research  questions. 
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D.  LITERATURE  REVIEW 


A.  INTRODUCTION 

This  chapter  begins  with  a  review  of  current  literature  in  the  area  of  performance 
measurement  and  outcome  management.  Specific  focus  is  given  to  the  area  of 
performance  measurement.  Next  the  chapter  reviews  the  United  States  Naval  Academy’s 
strategic  plan  and  current  performance  measurement  techniques.  Finally,  the  chapter 
reviews  the  Naval  Academy’s  Plebe  Summer  program.  The  Plebe  Summer  review 
centers  on  the  goals,  objectives,  procedures,  and  curriculum  of  the  program  as  well  as 
current  performance  and  outcome  measuring  procedures. 

B.  PERFORMANCE  MEASUREMENT 

Leading  and  controlling  organizations  has  grown  more  difficult  for  a  number  of 
reasons.  The  size  and  complexity  of  organizations  has  increased.  Additionally,  the  rate 
of  technologic  change  has  also  accelerated.  These  changes  coupled  with  increases  in 
diversity  among  customers,  employees,  markets,  and  stakeholders  create  difficult 
management  and  control  obstacles.  Kaplan  and  Norton  (1996)  identify  the  necessity  for 
business  to  change  in  the  face  of  the  “Information  Revolution”: 

Companies  are  in  the  midst  of  a  revolutionary  transformation.  Industrial 
age  competition  is  shifting  to  information  age  competition.  During  the 
industrial  age,  from  1850  to  about  1975,  companies  succeeded  by  how 
well  they  could  capture  the  benefits  from  economies  of  scale  to  scope. 
Technology  mattered,  but,  ultimately,  success  accrued  to  companies  that 
could  embed  the  new  technology  into  physical  assets  that  offered  efficient, 
mass  production  of  standard  products  (p.  2). 

However,  they  say,  a  new  business  environment  has  emerged  and  this  new  environment 
poses  new  challenges: 

The  emergence  of  the  information  era,  however,  in  the  last  decades  of  the 
twentieth  century,  made  obsolete  many  of  the  fundamental  assumptions  of 
industrial  age  competition.  No  longer  could  companies  gain  sustainable 
competitive  advantage  by  merely  deploying  new  technology  into  physical 
assets  rapidly,  and  by  excellent  management  of  financial  assets  and 
liabilities  (Kaplan  and  Norton,  1996,  p.  3). 
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New  assumptions  that  provide  the  foundation  of  information  age  organizations  are 
provided  by  Kaplan  and  Norton  (1996):  integrated  cross  function  business  processes, 
integrated  links  to  customers  and  suppliers,  customized  products  to  various  customer 
segments,  global  scale  in  order  to  provide  adequate  returns,  innovation,  and  investment  in 
knowledge  workers  (p.4-5).  Many  organizations  have  recognized  these  new  challenges 
and  have  attempted  to  improve  their  processes  in  order  to  meet  them.  However  many 
organizations  are  not  succeeding  in  meeting  the  new  challenges.  Kaplan  and  Norton 
(1996)  offer  performance  management  as  a  solution. 

Many  of  these  improvement  programs  have  yielded  disappointing  results. 

The  programs  are  often  fragmented.  They  may  not  be  linked  to  the 
organization’s  strategy,  nor  to  achieving  specific  financial  and  economic 
outcomes.  Breakthroughs  in  performance  require  major  change,  and  that 
includes  changes  in  the  measurement  and  management  systems  used  by  an 
organization.  Navigating  to  a  more  competitive,  technological,  and 
capability-driven  future  cannot  be  accomplished  merely  by  monitoring  and 
controlling  financial  measures  of  past  performance  (p.6). 

Attempting  to  overcome  these  obstacles,  private  sector  organizations  continually 
search  for  management  systems  that  provide  an  economic  edge  over  competitors. 
Performance  measurement  and  management  is  one  such  system.  World-class 
organizations  use  performance  measurement  systems  to  determine  whether  they  are 
fulfilling  their  vision  and  meeting  their  customer- focused  strategic  goals  (NPR,  1997, 
p.13).  In  short,  performance  measurement  is  the  process  of  assessing  progress  toward 
organizational  goals,  including  the  efficiency  with  which  resources  are  changed  into 
outputs,  the  quality  of  outputs,  outcomes  achieved  and  the  overall  effectiveness  of  the 
organization’s  efforts  toward  their  mission  (NPR,  1997,  p.  4) 

In  1993,  President  Clinton  signed  into  law  the  Government  Performance  and 
Results  Act  (GPRA).  This  legislation  forced  federal  government  agencies  to  adopt  the 
practice  of  strategic  management.  The  act  lists  six  purposes  for  the  GPRA,  among  those 
listed  are: 


•  Improve  the  confidence  of  the  American  people  in  the  capability  of  the 
Federal  Government,  by  systematically  holding  Federal  agencies  accountable 
for  achieving  program  results. 
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•  Improve  Federal  program  effectiveness  and  public  accountability  by 
promoting  a  new  focus  on  results,  service  quality,  and  customer  satisfaction. 

•  Help  Federal  managers  improve  service  delivery,  by  requiring  that  they  plan 
for  meeting  program  objectives  and  by  providing  them  with  information  about 
program  results  and  service  quality. 

•  Improve  internal  management  of  the  Federal  Government  (USC,  1993,  p.  1). 

The  GPRA  requires  all  governmental  agencies  to  design  and  submit  strategic 
plans  to  the  President  and  the  Congress.  Strategic  plans  are  required  to  include  agency 
goals,  performance  objectives,  and  performance  metrics  to  indicate  success  or  failure  in 
meeting  agency  goals.  Each  governmental  agency  is  then  required  to  report,  to  the 
President  and  the  Congress,  annually  on  agency  performance  based  on  their  strategic 
plans  and  performance  objectives  (USC,  1993,  p.  1).  Therefore,  by  signing  the  GPRA 
President  Clinton  institutionalized  strategic  management,  performance  management,  and 
performance  measurement  throughout  the  federal  government. 

Performance  measurement  and  management  has  been  widely  studied  and  reported 
on.  In  general  they  have  each  received  very  positive  support  (Kaplan  and  Norton,  1996; 
Harbour,  1997;  Arveson,  1998a;  PEA,  1998;  Zigon,  1998;  NPR,  1999;  Belz,  1999;  Hatry, 
1997;  Kamensky,  2001).  Many  public  and  private  organizations  have  shifted  (or  are 
shifting)  to  management  systems  that  include  performance  measurement  techniques 
(PEA,  1998).  These  management  shifts  have  been  undertaken  in  order  to  better  control 
various  concerns  such  as  productivity,  efficiency,  profitability,  and  customer  and 
employee  satisfaction  (NPR  1997). 

However,  performance  measurement  has  not  received  universal  praise  (Scheirer, 
2000,  Perrin,  1998;  Arveson,  1998b,  1998c).  Most  of  the  concern  about  performance 
measurement  and  management  centers  on  the  fact  that  performance  measures  illustrate 
only  how  a  company  or  program  is  performing.  Performance  measurement  techniques 
cannot  indicate  why  a  company  or  program  achieves  its  level  of  performance.  The 
disconnect  between  performance  measurement  and  causal  forces  continues  to  be  a  source 
of  criticism  throughout  the  literature  (Schreier,  2000). 
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1. 


Goals  And  Objectives 


Performance  management  systems  are  designed  to  provide  information  required 
for  the  fundamental  decisions  and  actions  that  shape  and  guide  what  an  organization  is, 
what  it  does,  and  why  it  does  it.  Performance  measurement  is  a  tool  that  when  properly 
employed  focuses  organizational  attention  in  order  to  better  plan  and  execute  strategic 
actions.  The  strategic  focus  provided  by  performance  management  is  intended  to  result 
in  improved  organizational  outputs  and  outcomes  and  is  built  on  a  foundation  of 
performance  measurement.  NPR  (1997)  suggests  that  best-in-class  organizations  use 
measurement  information  to:  set  goals  and  standards;  detect  and  correct  problems; 
manage,  describe,  and  improve  processes;  and  document  accomplishments  (p.31). 

Performance  measurement  is  one  of  many  available  tools  that  organizations  use  to 
manage  their  processes  and  control  their  organizational  outputs  and  outcomes.  The  PEA 
(1998)  describes  how  organizations  exploit  performance  measurement  and  outcome 
management  to  assists  in  the  management  of  their  programs: 

One  of  the  hallmarks  of  leading- edge  organizations  -  be  they  public  or 
private  -  has  been  the  successful  application  of  performance  measurement 
to  gain  insight  into,  and  make  judgments  about  the  organization  and  the 
effectiveness  and  efficiency  of  its  programs,  processes,  and  people. 
However,  leading  organizations  do  not  stop  at  the  gathering  and  analysis 
of  performance  data;  rather  these  organizations  use  performance 
measurement  to  drive  improvements  and  successfully  translate  strategy 
into  action.  In  other  words,  they  use  performance  measurement  for 
managing  their  organizations  (p.  vii). 


The  National  Partnership  for  Reinventing  Government  (1997)  echoes  this  view: 


All  high-performance  organizations  are,  and  must  be,  interested  in 
developing  and  deploying  effective  performance  measurement  and 
performance  management  systems,  since  it  is  only  through  such  systems 
that  they  can  remain  high-performance  organizations  (p.  1). 


Additionally,  Hatry  and  Kopczynski,(1997)  say  that  not  only  performance  should  be 
measured.  They  express  a  need  to  also  measure  program  outcomes: 


12 


Developing  program  outcome  information  is  a  critical  step  in  developing  a 
high  quality  performance  measurement  system  for  programs  and  projects 
for  the  federal  government.  Good  program  management  requires 
collection  and  use  of  outcome  data  to  provide  guidance  for  improvement. 
Without  information  on  results,  managers  can  only  supervise  “inputs”  or 
monitor  processes.  Decisions  on  whether  the  program  is  actually  working 
well- or  needs  changing- are  made  in  the  absence  of  hard  data  on  actual 
outcomes  (p.  6). 

Organizations  must  first  recognize,  before  taking  action,  the  rationale  on  which 
performance  measurement  is  founded.  A  great  deal  of  the  research  postulates  why 
organizations  should  undertake  performance  measurement  (Averson,  1998a;  Czarnecki, 
1999;  Frost,  2000;  NPR,  1997,  1999;  PEA,  1998;  Frost,  2000).  There  are  four  basic 
reasons  most  organizations  undertake  performance  measurement: 

•  You  cannot  manage  what  you  cannot  measure.  In  the  absence  of  performance 
measurement  managers  cannot  set  goals,  provide  feedback,  or  communicate 
success. 

•  You  cannot  improve  what  you  cannot  measure.  Without  performance  data 
managers  cannot  evaluate  if  program  alterations  result  is  actual  program 
improvement. 

•  High  performance  teams  and  individuals  require  clear  goals.  Clear  goals  are 
required  so  excellent  performance  can  be  recognized  in  highly  diverse  and 
intricate  organizations. 

•  Reward  systems  require  metrics.  When  pay  or  rewards  are  based  on 
performance,  then  managers  require  some  way  of  knowing  when  reward  has 
been  earned  (Zigon,  1998,  p.l). 

Additionally,  the  National  Center  for  Public  Productivity  (1997)  identifies  other  benefits 
performance  measurement  provide  public  organizations  such  as  greater  accountability  of 
managers,  increase  in  service  to  public,  and  stimulation  of  public  participation  (NCPP, 
1997,  p.3). 

The  objective  of  performance  management  in  the  public  sector  can  thus  be 
defined  as  the  detailed  development  and  exploitation  of  performance  metrics  that 
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measure  the  success  of  organizational  long-term  and  short-term  strategic  goals  in  order  to 
increase  service  effectiveness.  As  performance  management  matures  and  becomes  more 
established  it  becomes  more  important;  thus  it  implies  measure- based,  strategy- driven 
organizational  control  (Frost,  2000). 

2.  Performance  Measurement  Terminology 

Before  getting  deeper  into  performance  measurement  and  performance 
management,  it  is  useful  to  define  the  many  terms  used  in  this  field.  The  literature  offers 
a  variety  of  similar  but  varying  definitions  for  many  of  the  performance  measurement 
terms  (NPR,  1997;  Hatry  and  Kopczynski,  1997;  Frost,  2000;  Harbour,  1997).  The 
following  definitions  will  be  used  in  this  research  to  standardize  the  terms: 

Performance  Measurement:  The  regular,  ongoing  measurement,  and  reporting 
on  important  performance  aspects  of  organizational  programs. 

Performance  Measure :  A  quantitative  or  qualitative  characterization  of 
performance. 

Benchmark:  What  outcome  indicators  or  performance  measure  information  is 
compared  to.  Benchmarks  provide  a  comparison  for  data.  Benchmarks  provide  the 
minimum  level  of  performance  or  change  acceptable. 

Performance  Management :  The  use  of  performance  measurement  information 
to  help  set  agreed-upon  performance  goals,  allocate  and  prioritize  resources,  inform 
managers  to  either  confirm  or  change  current  organizational  policies  and  procedures,  and 
report  the  success  in  meeting  organizational  goals. 

Inputs:  The  amount  of  organizational  inputs  applied  (i.e.  amount  of  funds,  time, 
or  man-hours). 

Outputs:  Amount  of  organizational  product  completed  (i.e.  work  processed,  test 
scores,  number  and  quality  of  products) 

Outcomes:  Not  what  organizations  do,  but  the  consequences  of  their  actions. 
Outcomes  happen  outside  the  organizations  such  as  to  customers  or  other  organizations 
whose  behavior  the  organization  hopes  to  affect.  Outcomes  are  classified  as  either 
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intermediate  or  end.  Intermediate  outcomes  take  actions  that  lead  to  end  outcomes.  End 
outcomes  are  the  final  desired  results  of  the  program’s  work. 

Outcome  Indicator:  This  is  not  the  same  as  outcomes.  Outcome  indicators 
specify  what  is  to  be  measured.  Specific  indicators  will  depend  on  the  type  of  data 
collection  used.  Outcome  indicators  identify  specific  numerical  values  that  indicate 
progress  toward  achieving  an  outcome  (such  as  percentage  or  ratio). 

Outcome  Measurement  Process:  The  process  for  selecting  outcome  indicators 
and,  subsequently,  regularly  obtaining  and  reporting  data  on  the  indicators.  Outcome 
indicators  are  required  for  every  significant  program  objective. 

Impact:  The  extent  to  which  a  program  actually  caused  an  outcome. 

3.  Best  Practices 

There  are  a  variety  of  methods  to  employ  the  principles  of  performance 
measurement  and  performance  management  (Kaplan  and  Norton,  1996;  Hatry  and 
Kopczynski,  1997;  NPR,  1997;  PEA.  1998).  Each  approach  possesses  its  own  unique  set 
of  strengths  and  weaknesses,  as  well  as  challenges  and  opportunities.  However,  there  are 
several  characteristics  of  successful  performance  management  systems  that  have  been 
identified  as  likely  to  result  in  success.  The  National  Partnership  for  Reinventing 
Government  (1997)  studied  and  reported  on  the  performance  management  operations 
among  32  “best- in-class”  organizations.  A  number  of  “best  practices”  were  identified. 
Because  world-class  organizations  use  performance  measurement  systems  to  determine 
whether  they  are  fulfilling  their  vision  and  meeting  their  customer- focused  strategic 
goals,  their  performance  measures  must  therefore  meet  the  following  criteria  (NPR,  1997, 
P-13): 

•  Ensure  narrow,  strategic  focus.  The  measures  and  goals  an  organization  sets 
should  be  narrowly  focused  to  a  critical  few.  It  is  neither  possible  nor 
desirable  to  measure  everything  (NPR,  1997,  p.13).  Attempting  to  measure 
too  much,  or  everything,  leads  to  data  overload.  In  this  situation  managers 
cannot  distinguish  important  from  irrelevant  data. 
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•  Measure  the  right  thing.  Before  deciding  on  specific  measures,  an 
organization  should  identify  and  thoroughly  understand  the  processes  to  be 
measured.  Then,  each  key  process  should  be  mapped  to  ensure  (1)  a 
thorough,  rather  than  assumed  understanding  of  the  process;  and  (2)  that  a 
measure  central  to  the  success  of  the  process  is  chosen.  In  some  cases, 
targets,  minimums,  or  maximums  are  defined  for  each  measure  (NPR,  1997, 
p.13).  This  begins  with  first  identifying  what  the  organization  must  do  well 
and  developing  a  measure  that  represents  this  process  (PEA,  1999,  p.15). 

•  Be  a  means,  not  an  end.  In  a  best-in-class  organization,  employees  and 
managers  understand  and  work  toward  the  desired  outcomes  that  are  the  core 
of  their  organization’s  vision.  They  focus  on  achieving  organizational  goals 
by  using  performance  measures  to  gauge  goal  achievement  but  do  not  focus 
on  the  measures  per  se.  Performance  measurement  is  thus  seen  as  a  means, 
not  an  end  (NPR,  1997,  p.  13). 

The  NPR  (1997),  from  their  research,  provides  public  sector  agencies  with  key 
strategies  to  implement  performance  management  systems  that  meet  the  GPRA 
requirements.  The  NPR  recommends  a  seven- step  process: 

1.  Establish  and  update  performance  measures  and  goals.  For  each 
organizational  goal  or  objective  performance  baseline,  targets,  and  measures  need  to  be 
established.  In  general  a  good  measure: 

a.  Is  accepted  by  and  meaningful  to  the  customer; 

b.  Tells  how  well  goals  and  objectives  are  being  met; 

c.  Is  simple,  understandable,  logical,  and  repeatable; 

d.  Shows  a  trend; 

e.  Is  unambiguously  defined; 

f.  Allows  for  economical  data  collection; 

g.  Is  timely  and  sensitive; 
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The  typical  definition  of  measures  includes  specific  goals,  data  requirements,  calculation 
methodology,  and  rationale  for  the  measure  (NPR,  1997,  p.  33-34). 

2.  Establish  accountability  for  performance.  Both  organizational  and 
individual  responsibilities  need  to  be  identified  for  the  performance  measures.  In  some 
organizations,  goal  targets  do  not  become  official  until  it  is  agreed  to  through  a 
negotiation  process  between  the  goal  owner  and  management.  This  ensures  a  high  degree 
of  integrity  in  the  process  and  the  people  involved  (NPR,  1997,  p.34). 

3.  Establish  data  collection  process.  Measures  must  be  timely,  easy  to 
implement,  and  clearly  defined.  Speed  is  essential  in  data  collection  and  distribution. 
Data  collection  should  not  be  overly  structured.  Management  should  define  who  is 
responsible  to  ensure  data  collection  is  collected  and  processed  accurately  and  reliably. 
Automation  should  be  used  whenever  possible  in  order  to  reduce  burden  on  the 
workforce  (NPR,  1997,  p.  35). 

4.  Establish  data  analysis  and  reporting  process.  Various  processes  can  be 
used  to  analyze,  validate,  and  report  measurement  results.  The  most  common  processes 
are  operations  research,  quality  control,  and  statistical  and  qualitative  analysis  (NPR, 
1997,  p.  36). 

5.  Evaluate  and  use  performance  information.  Performance  information 
must  be  formally  reviewed  and  action  upon  to  improve  or  simplify  process.  This 
provides  management  feedback  for  adjusting  future  performance  plans  and  resources. 
The  June  2,  1997  Federal  Times  cites  several  examples  of  agencies  improving  and 
refining  performance  measures: 

a.  Instead  of  counting  the  number  of  forecasts  it  makes,  the  National 
Weather  Service  measures  the  warning  time  given  to  the  public  before  severe 
weather.  The  lead-time  before  tornadoes  increased  from  seven  minutes  to  nine 
minutes. 

b.  Instead  of  tracking  the  response  time  to  questions  about  veterans’ 
eligibility  for  burial  on  its  grounds,  the  National  Cemetery  System  is  trying  to 


17 


improve  the  scheduling  of  burial  services.  Its  goal  is  to  finalize  a  date  within  two 

hours  of  receiving  a  request.  (NPR,  1997,  p.  36) 

6.  Report  performance  to  customers  and  stakeholders.  Data  should  be 
reported  and  performance  explained  internally  and  externally  across  the  organization. 
Reporting  can  be  done  using  paper  documents,  internet,  television,  annual  reports,  or 
other  multimedia  methods  (NPR,  1997,  p.36). 

7.  Repeat  the  process.  Sharing  performance  information  with  management, 
customers,  and  stakeholders  facilitates  deeper  understanding  among  all  parties.  Each 
stakeholder  can  provide  positive  information  to  update  and  revise  performance 
measurement  systems  for  improvement.  Therefore,  the  systems  are  not  static  and  must 
constantly  renew  themselves  (NPR,  1997,  p.  37) 

4.  Misuses  of  Performance  Measurement 

Perrin  (1998)  provides  valuable  balance  to  the  enthusiastic  support  performance 
measurement  programs  have  received.  His  article  provides  insights  to  many  of  the 
limitations  of  performance  measurement.  He  says,  “Performance  measurement 
programs,  by  themselves,  are  not  appropriate  for  assessing  outcomes,  for  determining 
future  directions  or  for  resource  allocation.  They  can,  however,  be  one  important 
component  of  a  comprehensive  evaluation  strategy”  (p.  1).  Moreover,  he  says  that 
performance  measurement  programs  are  not  living  up  to  their  billing  as  a  means  to  ensure 
focus  on  results,  produce  greater  accountability,  or  providing  control  over  expenditures 
(Perrin,  1998,  p.  2). 

Perrin  (1998)  identifies  eight  limitations  of  applying  performance  measures  to 
determine  program  performance: 

1.  Varying  interpretations  of  the  “same  terms  and  concepts:  Regardless  of 
how  clearly  defined  performance  measures  seem  they  can  be  interpreted  differently 
among  individuals.  This  inconsistency  among  staff  can  result  in  inaccuracies  that  make 
measures  meaningless  in  the  long  term  (Perrin,  1998,  p.  4) 

2.  Goal  Displacement:  When  measures  become  the  objective,  they  result  in 
“goal  displacement”.  This,  in  turn,  emphasizes  the  wrong  activities  without  improving 
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actual  outcomes.  The  Viet  Nam  body  count  serves  as  a  classic  example  of  how  too  great 
a  focus  on  measures  reduced  overall  mission  effectiveness  (Perrin,  1998,  p.  5). 

3.  Use  of  meaningless  or  irrelevant  measures:  Performance  measures  are  a 
simplification  of  a  complex  system. 

Attempting  to  reduce  a  complex  program  or  social  intervention,  such  as 
initiatives  in  child  welfare,  economic  development,  or  health  promotion,  to 
a  small  number  of  quantitative  indicators  can  disguise  and  mislead  rather 
than  inform  what  is  really  happening.  Thus  performance  measures  can 
confuse  performance  “indicators”  with  the  underlying  reality  (Perrin, 

1998,  p.  6). 

4.  Cost  savings  vs.  Cost  shifting:  Performance  measures  are  unable  to 
indicate  underlying  realities  of  a  program.  Therefore,  they  are  unable  to  indicate  if  a 
performance  measures  are  indicating  true  savings  or  simply  shifting  costs  to  the  future. 

5.  Critical  subgroup  differences  disguised  by  misleading  aggregate 

indicators:  Performance  measures  often  do  not  separate  larger  groups  and  apply 

performance  data  inaccurately  across  then  group  as  a  whole.  This  will  tend  to  disguise 
indicators  that  could  be  more  accurate  if  disaggregated  by  race,  gender,  age,  education, 
etc  (Perrin,  1998,  p.  7) 

6.  Limitations  of  objective-based  approaches  to  evaluation:  Performance 
management  programs  do  not  take  a  situational  approach  while  applying  performance 
data.  Effective  programs  may  be  penalized  by  a  performance  management  program  for 
responding  to  changes  in  its  environment  that  have  not  been  updated  in  the  overall 
performance  management  program  (Perrin,  1998,  p.8). 

7.  Useless  for  decision  making  and  resource  allocation:  Although  one  of  the 
primary  justifications  for  performance  measurement  is  its  ability  to  provide  for  more 
informed  decision  making,  by  itself,  it  is  useless  for  this  purpose.  Performance  measures 
can  indicate  what  is  happening  in  a  program  but  it  cannot  indicate  why.  Therefore, 
managers  must  investigate  further,  and  not  rely  on  performance  measures  alone,  when 
deciding  on  actions  and  resource  allocations  (Perrin,  1998,  p.8) 
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8.  Less  focus  on  outcomes:  Performance  measurement  leads  to  less  rather 
than  more  focus  on  outcomes,  innovation,  and  improvements.  The  results  obtained  from 
narrow  focus  on  numbers  limits  staff  empowerment,  innovation,  and  self-evaluation.  Too 
much  focused  placed  on  data  collection  leads  to  impaired  performance  and  a  reluctance 
to  admit  that  improvement  is  required  (Perrin,  1998,  p.  8). 

C.  NAVAL  ACADEMY  STRATEGIC  PLANNING  AND  PERFORMANCE 
MEASUREMENT 

1.  Naval  Academy  Strategic  Plan 

The  United  States  Naval  Academy  (USNA)  is  the  premier  source  of  officers  for 
the  Naval  Service.  It  provides  a  four-year  total- immersion  program  where  a  strong, 
balanced  academic  program,  focusing  on  the  education  needs  of  the  Naval  Service  is 
superimposed  on  a  strict,  professional  military  training  environment  (USNA,  1997,  p.l). 
Throughout  the  more  than  150- year  history  of  the  Naval  Academy  the  institution  has 
evolved  from  a  trade  school  focused  on  the  arts  of  seamanship  to  a  highly  regarded 
learning  institution.  The  Naval  Academy  today  provides  academic  majors  in  19  areas 
including  Engineering,  Sciences,  Humanities,  and  Economics. 

In  response  to  the  GPRA  the  USNA  developed  and  published,  in  1998,  its  first 
strategic  plan  to  address  its  mission  and  vision.  The  strategic  plan  has  been  consistently 
updated  yearly  since  its  publication.  The  plan  is  designed  to  avoid  mission  drift,  maintain 
appropriate  balance  between  academic,  professional,  and  athletic  programs,  and  provides 
the  foresight  and  focus  to  make  decisions  that  will  benefit  the  Brigade  of  Midshipmen 
and  the  Naval  Academy  for  10  years  (USNA,  2001c,  p.  2). 

The  USNA’s  strategic  plan  has  a  pyramid  structure  as  shown  in  Figure  1.  The 
principal  element  of  the  plan  is  the  Naval  Academy’s  vision.  The  vision  of  the  strategic 
planners  was  to  produce  a  Naval  Academy  that  would: 

Provide  leaders  of  great  character,  competence,  vision  and  drive  to 
transform  the  Navy  and  Marine  Corps  and  serve  the  nation  in  a  century  of 
promise  and  uncertainty  (USNA,  2001c,  p.  4) 


Supporting  the  vision  is  the  Naval  Academy’s  mission: 
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Develop  midshipmen  morally,  mentally,  and  physically  and  to  imbue  them 
with  the  highest  ideals  of  duty,  honor  and  loyalty  in  order  to 


Figure  1:  Naval  Academy  Strategic  Plan  Structure  [From:  USNA,  2001c.  p.  3-6] 

provide  graduates  who  are  dedicated  to  a  career  of  naval  service  and  have 
potential  for  future  development  in  mind  and  character  to  assume  the 
highest  responsibilities  of  command,  citizenship,  and  government 
(USNA,  2001c,  p.  3). 

To  support  the  vision  and  mission,  the  USNA  strategic  plan  employs  eight 
institutional  focus  areas.  Each  focus  area  is  designed  to  address  a  different  aspect  of  the 
Naval  Academy  and  is  described  below. 

•  Academic  Excellence  will  focus  on  instructional  and  learning  philosophies 
designed  to  promote  critical  thinking  and  lifelong  learning.  Moreover,  this 
focus  area  is  designed  to  provide  faculty  and  staff  resources  to  become  and 
remain  leaders  in  their  respective  disciplines. 

•  Admissions  Excellence  will  focus  Academy  efforts  to  attract  the  best  qualified 
men  and  women  willing  to  accept  the  challenges  of  the  naval. 
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•  Effective  Communications  is  designed  to  provide  Academy  programs  and 
support  structures  that  will  give  midshipmen  outstanding  written  and  verbal 
communication  skills.  Also,  it  is  designed  to  give  midshipmen  an  opportunity 
to  practice  open,  honest,  and  respectful  communication  skills. 

•  Character  Building  focuses  on  developing  midshipmen  who  are  committed  to 
selfless  service,  high  moral  standards,  ethical  behavior,  and  inculcated  with 
the  Naval  Service’s  core  values  of  honor,  courage,  and  commitment. 

•  Leadership  and  Professional  Excellence  will  prepare  midshipmen  for  the 
challenges  of  fleet  service  in  leadership  roles.  It  will  instill  a  deep  respect  for 
the  Constitution  and  support  the  commitment  to  high  moral  standards. 

•  Physical  Fitness  is  intended  provide  midshipmen  exemplary  programs 
designed  to  physically  challenge  midshipmen  and  promote  lifelong  physical 
fitness. 

•  Naval  Heritage  is  a  focus  area  intended  to  imbue  midshipmen  with  an 
appreciation  and  respect  of  selfless  service  that  are  representative  of  naval 
heritage. 

•  Academy  Quality  of  Life  receives  focus  to  ensure  a  high  quality  of  life  for 
midshipmen  and  staff.  This  focus  area  is  also  intended  to  provide  programs 
and  facilities  that  enrich  the  cultural,  recreational,  and  spiritual  experiences  for 
the  various  elements  of  the  Naval  Academy  community  (USNA,  2001c,  p.  9). 

The  foundation  of  the  USNA  strategic  plan  is  a  constantly  updated  set  of  tactical 
and  strategic  initiatives.  These  initiatives  are  designed  and  implemented  to  improve 
current  Academy  facilities  and  programs  or  to  create  new  programs  where  none 
previously  existed.  As  the  primary  source  of  officers  for  the  Navy  and  Marine  Corps  the 
Naval  Academy  has  set  very  high  quality  goals  for  its  graduates.  The  Academy  strategic 
plan  provides  the  following  list  of  graduate  attributes: 

•  Prepared  to  lead  in  combat; 


22 


•  Courageous  leaders  who  take  responsibility  for  their  personal  and  professional 
decisions  and  actions; 

•  Role  models  of  ethical  behavior  and  moral  conduct; 

•  Exemplars  of  academic,  technical  and  tactical  competence; 

•  Individuals  with  a  passion  and  commitment  to  lifelong  learning  and  physical 
fitness; 

•  Highly  effective  communicators; 

•  Leaders  who  recognize  and  value  individual  excellence  regardless  of  gender 
or  cultural  and  ethnic  background; 

•  Able  to  understand  and  integrate  geopolitical  complexities  in  their  decision 
making  across  the  spectrum  of  military  operations; 

•  Patriots  who  epitomize  the  rich  heritage,  honor  and  traditions  of  the  Navy, 
Marine  Corps,  and  our  country  (USNA,  2001d,  p.  1) 

2.  Naval  Academy  Performance  Measurement 

The  Naval  Academy  is  currently  using  its  strategic  plan  as  the  underpinning  to 
plan  and  measure  the  development  of  midshipmen  who  exemplify  the  previous  nine 
qualities.  This  section  reports  the  current  performance  measurement  used  by  the  Naval 
Academy  to  monitor  its  progress  and  processes  in  achieving  its  vision  and  mission. 

Two  theses  were  written  in  1999  which  investigated  performance  measurement 
techniques  used  at  the  Naval  Academy  (Belz,  1999;  Boone,  Hagen,  and  Utrotska,  1999). 
Both  theses  have  two  aspects  in  common.  Both  investigated  the  means  by  which  Naval 
Academy  Company  Officers  evaluated  midshipmen  development  and  both  used  Chang 
and  DeYoung’s  (1995)  Measurement  Linkage  Model  as  the  basis  of  their  investigation. 

Each  thesis  identified  the  Naval  Academy’s  mission  as  the  source  of  three  key 
results  areas:  the  moral,  mental,  and  physical  development  of  midshipmen.  However, 
Boone  et  al.  added  an  additional  key  results  area:  professional  development.  Belz  and 
Boone  et  al.  interviewed  Company  Officers  to  obtain  key  indicators  for  each  key  results 
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areas,  analyzed  the  information  obtained  using  Chang  and  DeYoung’s  model,  and  then 
reported  their  findings.  Figure  2  lists  the  key  results  areas  and  key  indicators  for  each 
area  listed  by  Belz  and  Boone  et  al. 

The  theses  provide  insight  into  the  use  of  performance  measurement  and 
management  techniques  used  at  the  Naval  Academy.  The  performance  measurement 
process  at  the  Naval  Academy  primarily  focuses  on  three  areas:  1)  academic 
performance,  2)  physical  fitness,  and  3)  honor  and  conduct.  Data  is  collected,  stored,  and 
retrieved  using  the  Midshipman  Information  Data  System  (MIDS).  The  MIDS  system, 
however,  does  not  provide  any  analysis  function.  It  is  used  only  as  a  database  and 
conduit  of  raw  information. 

These  two  theses  concentrate  on  the  performance  measurement  procedures 
applied  by  Company  Officers  to  assess  performance  on  a  company- wide  scale.  They  do 
not  represent  performance  measurement  of  Naval  Academy  programs.  No  research  has 
been  conducted  focusing  on  all-encompassing  performance  management  at  the  Naval 
Academy  or  performance  measurement  applied  to  individual  programs  or  curriculum 
such  as  Plebe  Summer. 

Surveys  are  an  important  performance  measurement  and  outcome  measurement 
tool  (Hatry  and  Kopczski,  1997,  p.  41).  The  Naval  Academy  annually  conducts  two 
important  opinion  surveys  of  midshipmen.  First  is  the  honor  survey.  This  survey  collects 
information  pertaining  to  midshipmen  perceptions  of  the  honor  concept  and  its 
applicability  to  the  Academy.  Second  is  midshipman  summer- training  survey.  This 
survey  collects  midshipmen  feedback  concerning  the  usefulness  and  applicability  of 
training  received  during  summer  sessions.  Summer  training  includes  professional 
practical  education  onboard  ships,  air  squadrons,  or  United  States  Marine  units. 

D.  FOURTH  CLASS  INDOCTRINATION  (PLEBE  SUMMER)  PROGRAM 

1.  Introduction 

Fourth  Class  Indoctrination  (Plebe  Summer)  program  is  the  first  phase  in  the 
Naval  Academy’s  year-long  Fourth  Class  Midshipman  Development  System  (USNA 
2001a,  p.  1).  The  second  phase  of  Fourth  Class  Development  System  is  the  Fourth  Class 
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Academic  Year  that  focuses  on  training,  vice  indoctrination  which  is  the  focus  of  Plebe 
Summer.  The  two  phases  together  are  designed  to  complement  each  other  and  ultimately 
lead  to  the  transition  of  midshipmen  from  civilian  and  enlisted  status  to  Naval  Academy 
Midshipmen  (USNA,  2001b,  p.  5).  This  section  reviews  the  purpose,  goals,  and 
objectives  of  the  Plebe  Summer  program  and  details  the  program  as  outlined  in  Naval 
Academy  Instructions. 


Belz 

Key  Results  Areas 

Moral  Development 

Key  Indicators 

None 

Mental  Development 

GPA 

Absences 

Physical  Development 

PE  Grades 

Physical  Readiness  Test  Scores 

Boone,  Hagen  &  Utroska 
Key  Results  Areas 

Moral  Development 

Key  Indicators 

Honor  Offenses/Counseling 

Conduct  Offenses/Demerits 

Community  Involvement 

Mental  Development 

GPA 

Number  of  Honor  Students 

Academic  Extracurricular  Activities 

Academic  Boards/Probation/UNSATs 

Study  Hours 

Physical  Development 

Physical  Readiness  Test  Scores 

PE  Failures/Grades 

Weight  and  Body  Fat  Standards 

Professional  Development  Absences 

Professional  Development  Grades 

Uniform/Room  Inspection  Grades 

4/C  Professional  Quiz  and  Boards\ 

Performance  Grades 

Figure  2:  Belz  and  Boone  et  al.  Key  Results  Areas  and  Key  Indicators 


Fourth  Class  Indoctrination  (Plebe  Summer)  serves  as  the  introduction  to  the 
Naval  Academy  for  midshipmen.  The  majority  of  midshipmen  enter  the  Naval  Academy 
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directly  from  high  school  and  are  experiencing  their  first  extended  period  away  from 
home.  However,  there  are  also  a  number  of  incoming  midshipmen  (approximately  20%) 
who  enter  the  academy  from  fleet  units  or  preparatory  schools.  Although  these 
midshipmen  possess  experience  in  military  culture,  Plebe  Summer  serves  as  an 
introduction  to  the  demands  of  the  Naval  Academy  for  them  as  well. 

Plebe  Summer  lasts  for  49  days  starting  with  Induction  Day  (I-Day)  and 
concluding  with  the  reform  of  the  brigade  and  the  start  of  the  academic  year.  Plebe 
Summer  encompasses  progressive  and  demanding  training  in  those  subjects  required  to 
produce  midshipmen  who  function  effectively  in  the  brigade,  are  trained  in  drill,  are 
physically  fit,  and  practice  those  personal  and  professional  traits  which  distinguish  them 
as  midshipmen  at  the  United  States  Naval  Academy  (USNA,  2001b,  p.  6).  Plebe  training 
is  not  designed  as  an  “initiation”.  It  is,  rather,  an  indoctrination  to  military  and  academy 
life  intended  to  prepare  fourth- class  midshipmen  for  the  rigorous  challenges  they  will 
face  during  the  four- year  academy  curriculum. 

The  Naval  Academy  lists  the  following  Plebe  Summer  objectives: 

•  Effect  the  transition  from  civilian  or  enlisted  to  officer  candidate  status. 

•  Indoctrinate  and  train  midshipmen  in  proper  military  discipline,  courtesy, 
requirements  and  traits  of  special  trust  and  confidence,  teamwork,  and 
personal  accountability. 

•  Provide  basic  training  in  seamanship,  marksmanship,  military  drill,  watch 
standing,  and  customs  and  traditions  of  the  Naval  service  (USNA,  2001b,  p. 
6). 

There  are  three  distinct  phases  of  Plebe  Summer:  the  Processing,  Forming,  and 
Training  phases.  As  midshipmen  progress  from  one  phase  to  another  they  face 
increasingly  stressful  and  challenging  environments.  The  following  reports  the  purpose 
and  primary  focus  of  each  phase. 

The  Processing  Phase  is  designed  to  facilitate  the  administrative  transition  from 
civilian  to  military  life.  This  includes  equipment  and  uniform  issue,  room  set-up,  and  all 
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the  necessary  administrative  transition  paperwork.  As  much  as  feasible,  the  schedule  is 
filled  with  military  instruction  in  a  low  stress  environment  (USNA,  2001b,  p.  8).  The 
Processing  Phase  lasts  for  four  days.  Subjects  focused  on  during  this  phase  include 
academic  testing,  introduction  to  religious  programs,  medical  physicals,  swim  testing, 
and  marksmanship  testing. 

Training- day  five  marks  the  beginning  of  the  Forming  Phase.  This  phase  is 
distinguished  from  the  Processing  Phase  by  an  increase  in  stress  and  pace  of  activities. 
The  purpose  of  this  segment  is  to  concentrate  on  basic  military  indoctrination,  instilling 
discipline,  developing  teamwork,  and  promoting  a  good  sense  of  Naval  Heritage  (USNA, 
2001b,  p.  8).  This  phase  is  designed  to  onto  the  midshipmen  a  foundation  of  base- level 
knowledge  and  spirit  that  will  be  used  in  the  last  phase  of  Plebe  Summer.  Important 
subjects  covered  during  this  period  include  close-order  drill,  character  development 
education,  introduction  to  Navy  rights  and  responsibilities,  and  professional  counseling. 

The  last  and  longest  phase  is  the  Training  Phase.  The  primary  objective  of  this 
phase  is  continued  progressive  development  through  demanding  training  and  follow-up  to 
ensure  the  incoming  Plebe  class  meets  the  personal  and  professional  standards  expected 
of  midshipmen  (USNA,  2001b,  p.  8).  Upon  completion  of  the  Training  Phase 
midshipmen  are  expected  to  meet  the  following  training  objectives: 

•  Know  all  basic  rates  (professional  knowledge  requirements) 

•  Properly  wear  all  uniforms 

•  Be  able  to  provide  directions  to  any  significant  location  on  campus 

•  Be  qualified  in  small  sailing  craft  and  small  arms 

•  Be  qualified  to  properly  stand  military  watches  to  include:  Company  Mate  of 
the  Deck  (CMOD),  and  Main  Office  Messenger  (MOM) 

•  Demonstrate  a  complete  working  knowledge  of  Midshipman  Regulations 

•  Achieve  at  least  a  grade  of  “C”  on  the  Physical  Readiness  Test 
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2.  Training  Topics 

Training  and  enculturation  is  the  focus  of  Plebe  Summer.  Therefore,  there  are 
specific  Naval  Academy  instructions  that  schedule,  guide,  and  control  the  training  and 
cultural  indoctrination  process  (USNA  1996;  USNA,  2001a;  2001b,  USNA,  2001f; 
USNA,  2001g).  There  are  eight  focus  areas  of  Plebe  Summer  training  (USNA  2001a). 
Focus  areas  are  distinguished  from  one  another  by  distinct  objectives.  Each  focus  area 
has  discrete  lesson  topics  associated  with  its  requisite  objectives: 

•  Command  Religious  Programs  is  an  introduction  to  worship  opportunities  at 
the  Academy  and  provides  time  for  midshipmen  to  meet  with  Academy 
chaplains.  Additionally,  chaplains  meet  with  midshipmen  in  small  groups  to 
discuss  issues  such  as  adjustment  to  USNA  and  interpersonal  relationships. 

•  Physical  Education  training  is  comprised  of  two  parts,  physical  conditioning, 
testing,  and  training  and  introduction  to  various  sports  opportunities. 

•  Academic  focus  area  provides  midshipmen  with  an  introduction  to  their 
academic  careers  at  the  Academy.  It  includes  a  series  of  placement  exams, 
advising  sessions,  and  curriculum  briefs. 

•  Supply  objectives  include  all  equipment  and  gear  logistics.  Therefore,  Supply 
is  referred  to  as  activities  rather  than  training  or  instruction. 

•  Administrative  focus  area  provides  time  periods  to  fill  out  all  required 
paperwork  and  general  administration  following  induction. 

•  Medical/Dental  provides  time  for  initial  examinations.  Additional  training  is 
provided  in  first  aid,  smoking  cessation,  drug  and  alcohol  abuse,  and  personal 
hygiene. 

•  Professional  Development  is  designed  to  introduce  midshipmen  to  various 
professional  skills.  These  include  sailing,  damage  control,  career  information, 
and  visual  signals  training.  Midshipmen  are  provided  training  in  three  levels 
of  small  sailing  craft;  knockabouts,  lasers,  and  Navy  44’s. 
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•  Commandant  of  Midshipman  time  is  the  largest  and  most  diverse  focus  area. 
It  serves  as  a  “catch-all”  for  topics  not  associated  with  the  previous  seven 
focus  areas.  Topics  covered  in  this  focus  area  include  Honor  Training, 
Military  History,  Close-order  drill,  character  development,  and  others. 
(USNA,  2001a,  p.4-20) 

Appendix  A  lists  the  lesson  topics  and  activities  for  each  focus  area.  Topics  in  each 
focus  areas  are  presented  to  midshipmen  in  an  assortment  of  formats  also  listed  in 
Appendix  A.  The  following  is  a  description  of  training  formats: 

•  Instruction  (Inst)  is  a  period  of  training  for  which  specific  terminal  training 
objectives  exist.  Upon  completion  of  an  instruction  period,  plebes  will  have 
gained  a  new  skill  or  knowledge  that  is  testable  at  some  future  time. 

•  Brief  (Brief)  is  a  period  of  instruction  for  which  no  terminal  training 
objectives  exist.  Although  information  gained  will  be  useful  at  some  future 
time,  information  is  not  testable.  Examples  are  Superintendent  and 
Commandant  of  Midshipmen  briefs. 

•  Administrative  Event  (AdmnEvnt)  is  a  period  of  time  required  for 
administration  of  regimental  requirements  such  as  haircuts  and  equipment 
issue.  No  training  objectives  are  associated  with  these  events 

•  Operational  Events  (OpEvnt)  are  required  for  regiment -wide  operations. 
Examples  are  visits  to  the  Holocaust  Museum  and  Baltimore  Orioles  baseball 
game.  No  training  objectives  are  associated  with  these  events. 

•  Examinations  (Exam)  are  used  to  determine  skill  or  knowledge  level  of 
midshipmen.  These  time  periods  are  used  to  evaluate  training  effectiveness. 
(USNA,  2001a,  p.  3) 

The  majority  of  training  is  in  the  form  of  Squad  Leader  Instruction  (SLI)  periods. 
Plebes  receive  twenty- two  hours  of  SLI  as  outlined  in  Appendix  B.  These  instructional 
periods  present  Plebes  with  information  to  facilitate  the  transition  from  civilian  to 
military  life  (USNA,  2001a,  p.53). 
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3. 


Plebe  Performance  Evaluation 


Plebe  development  is  measured  through  written  examination,  verbal  examination, 
inspections,  physical  fitness  tests,  and  practical  application.  Plebes  must  pass  written 
examinations  in  the  subjects  of  nautical  rules  of  the  road,  damage  control,  and  visual 
signaling.  A  score  of  70%  is  required  to  pass  written  exams.  Verbal  examination  are 
conducted  in  the  form  of  “rates”.  Plebes  are  assigned  professional  knowledge  that  must 
be  acquired  according  schedule.  They  are  verbally  quizzed  (rated)  by  instructors  to 
assess  their  level  of  knowledge.  If  plebes  fail  to  demonstrate  assigned  professional 
knowledge  they  receive  additional  training  and  are  immediately  re-tested. 

Plebes  are  inspected  often  during  Plebe  Summer.  Primarily  inspections  are 
conducted  on  uniform  appearance  and  room  cleanliness.  Failure  in  any  inspection 
receives  immediate  remediation  and  correction.  (USNA,  2001b,  p.  5-31) 

Physical  fitness  training  and  testing  are  a  large  part  of  Plebe  Summer.  The 
Physical  Readiness  Test  (PRT)  is  conducted  three  times  during  Plebe  Summer;  each 
plebe  must  participate  in  an  initial,  intermediate,  and  final  PRT.  The  PRT  consists  of 
three  elements:  1)  number  of  pushups  in  two  minutes,  2)  number  of  sit-ups  in  two 
minutes,  and  3)  timed  1  Vi  mile  run.  Plebes  failing  to  pass  the  initial  or  intermediate 
PRT’s  receive  remediation  throughout  Plebe  Summer  (USNA,  2001b,  p.22). 

Practical  application  of  training  received  in  weapons,  sailing,  and  visual  signals 
are  required  during  Plebe  Summer.  Plebes  receive  the  opportunity  to  qualify  in  small 
sailing  craft  and  weapons  (USNA,  2001a,  p.  32)  Additionally,  Plebes  must  qualify  a 
professional  watch,  the  Midshipman  Mate  of  the  Deck. 

Plebes  receive  formal  evaluations  from  midshipman  detailers  twice  during  Plebe 
Summer.  Counseling  sheets  are  retained  by  squad  leaders  to  record  any  item  considered 
significant  in  the  plebe’s  development  (USNA,  2001a,  p  5-25).  Upper-class  midshipmen 
provide  evaluation  of  the  Plebes  in  the  following  areas:  personal  appearance,  military 
bearing,  attitude,  professional  knowledge,  self-confidence,  watch  standing,  military  drill, 
room  appearance,  and  conduct.  Plebes  receive  a  subjective  grade  of  excellent,  average, 
or  below  average  in  each  area.  Also,  academic  and  military  performance  tests  are 
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recorded  on  plebe  evaluations.  In  areas  where  Plebes  are  evaluated  as  below  average, 
remediation  is  provided  (USNA,  2001a,  p.  5-25). 

E.  CHAPTER  SUMMARY 

This  chapter  has  reviewed  current  literature  addressing  performance  measurement 
and  management  and  its  potential  benefits  in  assisting  organizations  to  change  in  the  face 
of  diverse  cultural  and  technological  advance.  The  purpose,  goals  and  objectives,  best 
practices,  and  terminology  of  performance  management  were  reviewed.  The  chapter  also 
reviewed  the  1993  Government  Performance  and  Results  Act  and  how  the  act 
institutionalized  performance  management  in  the  federal  government. 

Although  most  literature  provides  positive  support  for  performance  measurement 
and  management,  the  chapter  reviewed  some  possible  drawbacks  and  pitfalls  to 
performance  management  that  organizations  must  take  into  account  performance 
measurement  programs  are  developed. 

The  United  States  Naval  Academy  is  subject  to  the  GPRA  and  has  developed  its 
own  strategic  plan.  The  plan  was  introduced  in  this  chapter  as  well  as  its  purpose, 
objectives,  and  goals. 

Finally,  the  Naval  Academy’s  Plebe  Summer  program  was  introduced.  The 
introduction  included  the  purpose,  goals,  objectives,  and  training  curriculum  of  plebe 
summer  as  well  as  Plebe  evaluation.  The  remaining  chapters  of  this  thesis  will  more 
deeply  examine  how  the  Naval  Academy  uses  performance  management  to  evaluate  the 
effectiveness  of  the  Plebe  Summer  program  in  meeting  its  objectives. 
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HI.  PERFORMANCE  MEASUREMENT  MODEL 


A.  INTRODUCTION 

There  are  numerous  performance  measurement  models.  This  chapter  describes 
Hatry  and  Kopczynski’s  (1997)  Guide  to  Program  Outcome  Measurement  for  the 
Department  of  Education  This  model  provides  procedures  and  recommendations  for 
developing  an  outcome  measurement  process  for  individual  educational  programs  (Hatry 
and  Kopczynski,  1997,  p.  1).  The  focus  of  the  model  is  on  developing  a  system  that 
identifies  and  measures  outcome  indicators  rather  than  other  performance  measures  such 
as  goals,  outputs,  or  processes: 

Without  information  on  results,  managers  can  only  supervise  “inputs”  or 
monitor  processes.  Decisions  on  whether  the  program  is  actually  working 
well  -  or  what  needs  changing  -  are  made  in  the  absence  of  hard  data  on 
actual  outcomes  (Hatry  and  Kopczynski,  1997,  p.  1). 

Furthermore,  the  philosophical  premise  of  the  model  is  that  outcome  measurement  has 

four  basic  applications  (1997,  p.  2): 

•  First  and  foremost  outcome  information  should  help  program  managers  and 
their  staff  track  how  their  programs  are  performing.  That  information,  in  turn, 
assists  program  managers  and  guides  improvement  efforts.  The  information 
should,  for  example,  indicate  where,  when  and  under  what  conditions 
outcomes  appear  to  be  satisfactory  or  not  satisfactory. 

•  Second,  outcome  information  can  be  useful  for  developing  and  justifying 
budgets  and  for  formulating  recommendations  as  to  needed  legislation  and 
policy. 

•  Third,  outcome  information  is  used  by  governing  organizations  in  establishing 
accountability  of  programs  for  program  quality  and  outcomes. 

•  Lastly,  outcome  information  can  be  used  to  help  communicate  with  and 
inform  customers  and  the  public  at  large  as  to  the  extent  to  which  progress  is 
being  made. 
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The  model’s  structure  is  simple.  It  consists  of  twelve  steps  divided  into  two 
categories:  three  preliminary  steps  and  nine  process/implementation  steps.  The  intention 
of  the  model  is  to  assist  managers  in  developing  high  quality  performance  measurement 
systems  or  improve  the  ones  already  in  place  (1997,  p.  2). 

B.  PRELIMINARY  STEPS 

Three  preliminary  steps  should  be  taken  prior  to  commencing  the  tasks  of 
developing  an  outcome  measurement  process  (Hatry  and  Kopczynski,  1997,  p.  11-12). 

1.  Determine  the  scope  of  the  program  included  in  the  measurement  process.  If 
there  is  more  than  one  activity  associated  with  the  program  then  program 
managers  must  determine  which  activities  will  be  included  in  the 
measurement  process.  Moreover,  many  programs  contain  activities  that 
overlap  with  other  programs.  Managers  should  decide,  prior  to  developing  a 
measurement  process,  which  overlapping  programs  will  included  in  their 
measurement  processes 

2.  Secure  top-level  support  for  the  outcome  measurement  effort.  Strong 
commitment  is  necessary  from  senior  level  managers  in  order  to  insure 
adequate  resources  are  committed  to  develop,  implement,  and  operate  the 
measurement  process. 

3.  Establish  a  working  group  to  oversee  development  of  the  process.  The 
working  group  should  include  stakeholders  from  every  aspect  of  the  program. 
Along  with  the  program  manager,  important  stakeholders  include: 

a.  Staff  and  other  employees 

b.  Representatives  from  related  programs 

c.  Technical  Experts  or  outside  consultants 

d.  Budget  personnel 

e.  Program  users 

These  preliminary  steps  ensure  a  strong  foundation  is  constructed  prior  to  building  an 
outcome  measurement  process.  Program  managers  should  not  omit  these  preliminary 
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steps  because  they  provide  the  initial  framework  for  the  development  and  implementation 
processes. 

C.  DEVELOPMENT  AND  IMPLENTATION  STEPS 

The  following  nine  steps  detail  the  necessary  actions  program  managers  must 
undertake  and  then  use  to  develop  and  implement  a  successful  outcome  measurement 
system  (Hatry  and  Kopczynski,  1997,  p.  19-87). 

1.  Identify  program  mission/objectives  and  customers:  The  first  step  is  precisely 
identifying  the  program  mission,  objectives,  and  customers.  It  begins  with 
developing  a  mission  statement  that  will  serve  as  a  starting  point  for 
identifying  the  outcomes  to  be  measured  and  the  specific  performance 
indicators  required.  The  mission  statement  should  describe  the  overarching 
results  the  program  is  designed  to  achieve  or  the  purpose  of  the  program  and 
have  the  following  characteristics: 

a.  General  statement  of  the  major  missions/purposes/objectives  that  the 
program  is  intended  to  achieve 

b.  The  statement  should  not  contain  numerical  targets.  Such  targets  will 
be  developed  later  in  the  development  process. 

c.  All  major  objectives  that  the  program  hopes  to  achieve  should  be 
identified 

d.  The  statement  should  include  words  that  explicitly  call  for  minimizing 
possible  negative  or  unintended  program  effects. 

Customers  should  be  identified  unless  this  is  obvious.  Questions  that  should  be  answered 
when  identifying  customers  include:  1)  who  benefits  from  the  program?  and  2)  who 
might  be  hurt  by  program  activities? 

2.  Identify  outcomes  that  should  be  monitored.  All  relevant  outcomes  should  be 
listed.  There  are  numerous  sources  that  identify  important  outcomes  and  are 
often  are  the  same  as  the  sources  that  identify  program  mission  and  objectives. 

a.  Legislation 
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b.  Mission  statements 


c.  Strategic  Plans 

d.  Direction  from  senior  leadership 

e.  Direction  from  individual  customers  or  focus  groups 

f.  Individual  employees  or  focus  groups. 

g.  Complaint  information 

h.  Other  similar  programs 

i.  Logic  models  or  outcome  sequencing  charts. 

3.  Select  outcome  indicators:  Program  outcomes  are  not  the  same  as  outcome 
indicators.  Every  listed  outcome  should  be  translated  into  one  or  more 
outcome  indicators  that  identify  specifically  what  is  to  be  measured.  Outcome 
indicators  identify  specific  numerical  or  qualitative  value  that  indicates 
progress  toward  or  away  from  achieving  an  outcome.  Good  quality  indicators 
meet  certain  criteria: 

a.  Relevance  to  mission  and  the  outcome  it  is  intended  to  measure. 

b.  Importance  of  what  it  measures.  Does  it  measure  some  important 
aspect  of  the  outcome? 

c.  The  extent  it  is  not  duplicated  by  other  indicators. 

d.  The  extent  that  program  has  influence  or  control  over  the  values  of  the 
outcome.  Although  programs  cannot  achieve  complete  control  over  an 
outcome,  as  long  as  the  program  is  expected  to  ultimately  have  some 
tangible,  measurable  effect  on  the  outcome,  the  indicator  should  be 
considered  for  inclusion. 

e.  Feasibility  and  cost  of  collecting  indicator  information.  It  should  be 
noted  that  sometimes  the  most  costly  indicator  provides  the  best 
quality  data  and  should  be  retained. 
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f.  Indicators  must  include  a  numerical  statement  such  as  “percentage”, 
“incidence”,  “proportion”,  or  “rate  of’. 

g.  Level  or  ability  of  customers  to  provide  relevant  feedback  relating  to 
outcomes. 

h.  Wording  of  indicators  must  be  specific.  For  example,  a  program  that 
wants  to  increase  the  “number  of  teachers  that  have  received 
significant  professional  development”  will  need  to  define  specifically 
what  is  meant  by  “significant”  in  order  to  be  able  to  measure  the 
outcome  indicator  in  a  meaningful  way. 

4.  Identify  data  sources  and  data  collection  procedures:  The  list  of  indicators  is 
not  complete  unless  data  collection  methods  are  included.  Four  major  sources 
of  educational  outcome  data  include  agency  records,  administered  tests, 
customer  surveys,  and  trained  observer  ratings.  These  four  sources  each  have 
strengths  and  weaknesses  when  serving  as  data  collection  methods. 
Furthermore  specific  requirements  to  ensure  each  source  is  effective  are  listed. 

Agency  records  are  a  good  source  of  data  due  to  their  accessibility  and  low  cost. 
Moreover,  the  procedures  associated  with  record  keeping  are  usually  familiar  to  program 
personnel.  However,  modifications  to  existing  record- collection  processes  may  be 
required  to  obtain  usable  outcome  data.  Additionally,  agency  records  alone  seldom 
provide  enough  information  on  program  quality  or  outcomes  and,  sometimes,  record 
information  may  have  to  be  merged  with  information  derived  from  the  records  of  other 
programs. 

Student  assessment  through  performance  testing  is  a  valuable  source  of  outcome 
data.  They  are  excellent  indicators  of  program  performance  and  outcome  data. 
Moreover,  grouped  test  data  can  be  categorized  by  various  breakout  characteristics  thus 
making  the  data  more  exploitable.  Testing  data  should,  however,  be  collected  in  a  timely 
fashion  so  that  program  outcome  is  best  represented  by  the  data. 

Customer  surveys  are  an  important  source  of  data.  Surveys  of  customers, 
systematically  conducted,  are  a  major  way  to  obtain  information  on  outcomes  such  as 
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behavior  and  satisfaction  with  program  characteristics.  This  information  often  is 
unavailable  from  any  other  source.  Moreover,  surveys  provide  direct  input  from 
customers  that  add  credibility  to  collected  data.  However,  surveys  are  difficult  to  design 
and  often  expensive.  Expertise  is  required  when  developing  a  survey  instrument  in  order 
to  ensure  accuracy  and  relevancy  of  data  gathered.  Additionally,  surveys  are  based  on 
respondents’  perceptions  and  memory.  These  weaknesses  can  cause  data  to  be  less 
convincing  than  other  data  source. 

Surveys  designed  to  collect  data  on  outcome  measurement  should  include 
questions  relating  to  outcomes  of  services,  amount  of  service  that  respondents  used, 
diagnostic  questions  (why  respondents  gave  particular  answers),  demographic  questions, 
and  requests  for  improving  services. 

Trained  observer  ratings  can  be  used  to  measure  outcomes  observable  by  the 
physical  senses.  With  this  method  personnel  are  trained  to  make  qualitative  ratings  of 
program  outcomes.  This  method  can  be  applied  in  situations  where  the  outcome  can  be 
measured  by  physical  observation  and  can  be  rated  on  a  scale  that  identifies  variations  in 
condition.  These  observations  can  have  a  high  degree  of  accuracy  and  reliability  if  the 
rating  system  is  clearly  defined,  adequate  training  of  observers  is  conducted,  and  if 
procedures  for  periodic  quality  reviews  are  developed  and  implemented. 

Like  the  other  methods  of  data  collection,  there  are  advantages  and  disadvantages 
to  this  method.  The  advantages  of  trained  observers  include  reasonably  accurate  rating  of 
conditions  that  are  otherwise  difficult  to  measure,  attaining  data  that  can  be  used  to 
allocate  resources,  and  observer  reports  are  usually  easy  to  present  and  understand. 
Disadvantages  include  die  labor  intensive  method,  ratings  require  monitoring  to  ensure 
observers  adhere  to  standards,  and  personnel  may  feel  uncomfortable  with  the  method. 

Outcome  indicators  should  not  be  considered  final  until  data  sources  and  data 
collection  procedures  are  chosen.  Data  collection  procedures  can  affect  the  specific 
outcome  indicator  used  to  measure  a  particular  outcome.  Program  managers  may 
consider  using  more  than  one  procedure  and  thus  more  than  one  indicator  to  track 
important  outcomes. 
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5.  Select  outcome  indicator  breakouts:  Program  data  alone  does  not  provide 
managers  with  usable  information;  intelligent  program  decisions  cannot  be  made  using 
raw  data.  Therefore,  all  data  collected  should  transformed  before  being  used  as  a 
decision  tool. 

The  first  transformational  step  is  disaggregation  of  data  (or  selecting  breakouts). 
Breakouts  permit  comparisons  among  groups  within  a  program.  They  also  should  be 
used  to  distinguish  important  groupings  that  have  quite  different  outcomes  from  other 
groups.  Therefore,  outcome  analysis  should  track  the  progress  being  made  separately  for 
each  group  to  provide  more  meaningful  information  on  what  is  happening  within  the 
program. 

Programs  should  consider  a  number  of  categories  when  deciding  what  breakouts 
are  most  useful  for  its  outcome  indicators  (Hatry  and  Kopczynski,  1997,  p.  53). 
Categories  of  breakout  information  include:  organizational  or  unit  objectives,  customer 
characteristics,  and  type  of  process  used  to  deliver  the  service.  These  categories  enable 
program  managers  to  separate  data  in  order  to  identify  where  their  programs  are  doing 
well  and  where  they  are  not. 

Separate  outcome  information  on  individual  projects  within  larger  programs  can 
be  quite  useful.  When  data  is  collected  from  large  programs  managers  of  individual 
projects  should  have  outcome  information  that  pertains  to  that  manager’s  own  area  of 
responsibility.  Outcome  data  that  lumps  together  outcomes  from  more  than  one  project  is 
not  likely  to  be  useful  to  managers  of  individual  projects.  Thus,  program  objectives 
should  be  identified  prior  to  data  collection  so  that  breakouts  can  be  more  easily 
identified  in  the  later  process. 

Breakouts  by  categories  of  customers  can  be  very  useful  in  providing  information 
to  program  personnel  about  the  extent  to  which  particular  categories  of  customer  services 
are  achieving  the  desired  outcomes  and  for  which  categories  desired  outcomes  are  not 
being  achieved.  Categories  in  this  breakout  include:  age,  gender,  race,  household 
composition,  etc.  Using  this  information,  breakout  characteristics  can  be 
comprehensively  tailored. 
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Relating  outcomes  to  the  type  and  magnitude  of  activities  being  supported  by  the 
program  is  likely  to  be  of  major  interest  to  program  managers.  Thus,  a  program  should 
consider  breaking  out  outcome  data  by  key  characteristics  of  the  projects  being  supported 
by  the  program.  Such  information  can  be  very  useful  in  distinguishing  the  more 
successful  from  less  successful  approaches. 

Many  programs  have  special  characteristics  of  relevance  to  them  that  do  not  fall 
into  one  of  the  previous  categories.  Therefore,  they  should  examine  each  of  its  outcome 
indicators  to  determine  what  breakouts  would  likely  help  them  identify  where  the  desired 
outcomes  are  successful  and  where  they  are  not. 

Breakout  selection  sho  uld  start  with  a  review  of  the  categories  listed  and  a  review 
of  outcome  indicators  to  determine  which  apply  to  the  program.  Sub  categories  should 
be  identified  within  each  major  category.  For  example,  by  what  age  level  groupings 
should  be  measured?  Under  20,  over  25,  between  17  and  22?  Or  should  data  be  sought 
by  individual  ages?  Which  subgroups  and  how  many  will  be  sought  will  be  determined 
in  part  by:  data  collection  procedures,  available  resources,  and  required  accuracy. 

6.  Compare  findings  to  benchmarks:  Benchmarks  provide  a  means  to 
determine  if  program  outcomes  are  good  or  bad.  This  step  identifies  the  types  of 
comparisons  (benchmarks)  useful  for  programs.  This  step  is  a  major  one  in  a  program’s 
analysis  of  the  findings  from  its  outcome  measurement  data  collection.  Major  types  of 
comparisons  include:  Previous  performance  that  may  indicate  improvement  over  time, 
performance  of  similar  units  that  may  provide  benchmarking  against  the  best,  pre¬ 
selected  targets,  and  outcomes  among  different  client  groups. 

Comparisons  against  previous  performance  are  the  most  common  type  of 
benchmark.  Current  performance  should  always  be  compared  to  that  of  previous 
reporting  periods,  regardless  of  the  data  collection  cycle. 

Programs  that  provide  essentially  the  same  service  are  likely  to  provide  useful 
benchmarks  to  managers.  Reporting  such  comparisons  can  also  have  motivational  value 
to  program  personnel. 
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Breakouts  by  various  customer  demographics  provide  ideal  benchmarking  data. 
Comparisons  should  be  made  to  indicate  whether  the  program  appears  to  be  more  or  less 
successful  with  certain  categories  of  customer/workload  than  with  others  such  as  males 
compared  to  females  (Hatry  and  Kopczynski,  1997,  p.  62). 

Pre-selected  targets  (goals)  are  extremely  useful  benchmarks.  They  are 
cornerstones  of  the  GPRA.  Programs  set  targets  and  report  achievement  or  non¬ 
achievement.  Programs  may  want  to  use  range  targets  rather  than  single  values.  For 
example  if  a  target  is  expressed  as  a  percentage,  the  target  might  be  the  range 
encompassing  the  target  plus  or  minus  five  percentage  points. 

When  an  alteration  of  program  delivery  occurs  outcome  measurement  can 
indicate  the  success  of  the  alteration.  This  benchmark  uses  previous  performance  as  a 
standard  to  measure  new  procedures.  Outcome  data  can  be  used  to  track  changes  in 
outcomes  from  before  the  change  to  the  outcomes  after  introduction  of  new  practices. 

7.  Pilot  test  the  procedures:  New  or  substantially  altered  data  collection 
procedures  should  be  pilot  tested  prior  to  full  implementation.  This  will  identify  and 
eliminate  any  problems  with  data  collection  procedures. 

Testing  should  look  for  problems  such  as:  unclear  definitions,  missing  data,  and 
confidentiality  problems.  Resolution  of  these  problems  can  include  procedure  alteration, 
deletion  of  outcome  indicator,  or  acceptance  of  diminished  accuracy. 

8.  Analyze  and  report  outcome  information-  Steps  5  and  6  discussed 
breakouts  and  comparisons  for  each  outcome  indicator.  These  choices  of  breakouts  and 
comparisons  will  likely  be  the  major  focus  of  a  program’s  analysis  effort  after  it  receives 
outcome  information. 

In  each  comparison  developed  in  step  6,  program  staff  should  identify  outcomes 
that  indicate  where  the  program  has  done  considerably  better  or  worse  than  anticipated. 
Furthermore,  staff  should  attempt  to  identify  why  this  occurred. 

How  programs  report  can  be  as  important  as  what  they  report.  There  are 
numerous  methods  to  report  outcomes: 
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a.  Present  actual  outcomes  to  targets  for  both  the  last  and  current 
reporting  periods. 

b.  Present  cumulative  values  for  a  year  rather  than  previous  reporting 
periods. 

c.  Present  responses  from  customer  surveys  with  responses  broken  out  by 
respondent  characteristics. 

d.  Other  methods  such  as  graphs,  bar  charts,  timelines,  and  maps. 

When  outcomes  are  reported  explanatory  information  should  be  included.  Such 
information  gives  the  program  the  opportunity  to  explain  significant  outcomes  that  were 
better  or  worse  than  expected. 

9.  Use  outcome  information-  It  is  important  to  remind  outcome  information 
users  that  outcome  data  don  not  tell  how  much  of  the  observed  outcomes  was  caused  by 
the  program  hand  how  much  by  other  factors.  Outcome  information  only  conveys 
whether  programs  are  meeting  outcome  objectives  or  not.  The  primary  use  of  outcome 
information  is  to  provide  regular  feedback  to  program  personnel.  There  are  various 
applications  to  the  feedback  of  outcome  information: 

a.  Motivate  personnel  to  improve. 

b.  Track  whether  actions  have  led  to  improvements 

c.  Identify  where  problems  exist 

d.  Communicate  with  and  inform  customers  and  the  public  about  the 
extent  to  which  progress  is  being  made. 

e.  Support  long-range  planning. 

f.  Serve  as  baseline  for  targets  and  goals. 

g.  Identify  need  for  program  personnel  training. 

D.  CHAPTER  SUMMARY 

This  chapter  reviewed  Hatry  and  Kopczynski’s  (1997)  Guide  to  Program 
Outcome  Measurement  for  the  U.S.  Department  of  Education  The  guide  provides 
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recommendations  for  development  of  an  outcome  measurement  process  for  individual 
education  related  programs. 

The  first  three  steps  are  preliminary  to  process  development.  These  preliminary 
steps  are  intended  to  gather  high-level  program  support,  determine  the  scope  of  the 
program  to  be  measured,  and  establish  a  working  group  to  oversee  the  development  of 
the  process.  The  guide  then  recommends  nine  process  and  implementation  steps: 

•  Mission,  objective,  and  customer  identification. 

•  Identify  outcomes  to  be  monitored 

•  Select  outcome  indicators 

•  Identify  data  sources  and  data  collection  procedures 

•  Select  outcome  indicator  breakouts 

•  Set  benchmarks  and  compare  findings  to  benchmarks. 

•  Pilot  test  data  collection  procedures 

•  Analyze  and  report  outcome  information 

•  Use  outcome  information 

Outcome  measurement  processes  developed  using  this  guide  provide  program 
managers  with  information  that  will  assist  them  in  tracking  the  performance  of  their 
program  and  identify  where  improvements  are  required.  Moreover,  outcome  information 
will  assist  managers  in  communicating  with  and  informing  customers  and  the  public  as  to 
the  success  of  education  related  programs. 

The  remaining  chapters  of  this  thesis  will  describe  the  research  methodology, 
analysis,  conclusions,  and  recommendations  for  the  Naval  Academy’s  Plebe  Summer 
program  performance  measurement  process  using  Hatry  and  Kopczynski’s  guide  as  a 
basis  of  analysis. 
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IV.  METHODOLOGY 


A.  INTRODUCTION 

The  purpose  of  this  research  is  to  analyze  the  performance  measurement  and 
outcome  management  procedures  applied  to  the  Plebe  Summer  program  at  the  United 
States  Naval  Academy.  The  research  methodology  is  divided  into  four  areas.  The  first 
area  consists  of  a  thorough  examination  of  current  performance  measurement  and 
outcome  management  practices.  The  Government  Performance  and  Results  Act  of  1993 
forced  United  States  governmental  agencies  to  apply  performance  measurement  practices 
in  planning  and  managing  their  programs.  The  Naval  Academy  is  part  of  the  Department 
of  Defense;  therefore,  it  is  subject  to  the  GPRA.  Due  to  the  requirements  of  the  GPRA 
investigation  of  current  performance  measurement  and  performance  management  is  the 
logical  first  step. 

The  second  area  of  research  methodology  consists  of  a  thorough  examination  of 
the  United  States  Naval  Academy  directives  and  military  instructions  governing  the 
management  of  Plebe  Summer.  This  area  provides  baseline  information  concerning  the 
performance  measurement  mechanisms  currently  used  by  the  Naval  Academy. 
Moreover,  this  area  also  examines  how  midshipman  performance  is  measured  and 
recorded  during  and  after  Plebe  Summer. 

Gathering  feedback  from  expert  Naval  Academy  personnel  is  the  third  area  of 
research  methodology.  In  order  to  effectively  identify  the  precise  methods  of  Naval 
Academy  performance  measurement  applied  to  Plebe  Summer.  Experts  selected  for  this 
research  included  only  personnel  in  key  management  positions  accountable  for 
determining  the  success  of  Plebe  Summer. 

The  final  research  area  uses  Hatry  and  Kopczynski’s  (1997)  Guide  to  Program 
Outcome  Measurement  to  evaluate  the  performance  measurement  and  outcome 
management  practices  applied  to  the  Plebe  Summer  program  by  the  Naval  Academy. 
This  guide  proposes  nine  development  and  implementation  steps  in  creating  an  effective 
performance  measurement  system  for  educational  programs. 
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B.  RESEARCHING  GENERAL  PERFORMANCE  MEASUREMENT 

Researching  Plebe  Summer  performance  measurement  begins  with  an  in-depth 
review  of  current  performance  measurement  literature.  This  management  tool  has 
applications  in  both  private  and  public  sectors.  Therefore,  not  all  theories  apply  to  the 
Academy’s  Plebe  Summer  program.  An  extensive  literature  review  concentrates  on 
current  theories  and  best  practices  applicable  to  public  sector  programs.  This  review 
examined  books,  journal  articles,  internet  articles,  encyclopedias,  and  governmental 
agency  reports.  The  following  subjects  concerning  performance  measurement  are 
researched  in  order  to  identify  how  best  to  apply  performance  measurement  to  the  Plebe 
Summer  program: 

•  Goals  and  objectives  of  performance  measurement  are  studied  in  order  to 
confirm  the  Plebe  Summer  program  is  indeed  an  activity  with  which 
performance  management  is  applicable. 

•  Performance  Terminology  is  researched  to  provide  a  consistent  vocabulary 
throughout  the  thesis. 

•  Best  practices  of  performance  measurement  are  researched  to  provide  positive 
examples  of  public  sector  employment  of  performance  management.  This 
area  identifies  how  public  sector  programs  work  to  meet  the  demands  of  the 
Government  Performance  and  Results  act  of  1993. 

•  Misuses  of  performance  measurement  are  researched  to  identify  areas  where 
performance  management  is  misapplied  and  could  possibly  lead  to  poor 
management  decisions 

C.  RESEARCHING  ACADEMY  PERFORMANCE  MEASURMENT 

Plebe  Summer  performance  measurement  procedures  are  derived  directly  from 
Naval  Academy  instructions  governing  Plebe  Summer.  Therefore,  a  detailed  review  of 
all  instructions  governing  Plebe  Summer  is  conducted.  Moreover,  because  Plebe 
Summer  instructions  should  support  larger  institutional  goals  a  review  of  the  Naval 
Academy  strategic  plan  is  conducted.  Additionally,  this  strategic  plan  is  reviewed  to 
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provide  information  on  the  Academy  leadership’s  attitude  and  goals  concerning  the  full 
development  of  midshipmen. 

Specific  governing  instructions  for  the  Plebe  Summer  program  are  researched  to 
identify  the  exact  goals,  objectives,  procedures,  and  performance  measurement  of  Plebe 
Summer.  These  instructions  provide  information  concerning  how  Plebe  Summer 
performance  is  recorded  and  examined  in  determining  the  level  of  success  or  failure  of 
the  program  in  meeting  program  objectives. 

D.  EXPERT  INTERVIEWS 

Expert  interviews  are  conducted  to  identify  the  performance  measurement 
procedures  used  by  Naval  Academy  personnel  responsible  for  planning,  executing,  and 
evaluating  Plebe  Summer.  Furthermore,  expert  interviews  are  used  to  further  elaborate 
the  intentions  of  Naval  Academy  instructions  and  as  well  as  identify  the  depth  to  which 
the  instructions  guide  the  performance  measures  of  Plebe  Summer. 

Interviews  are  conducted  with  six  experts  that  cover  three  levels  of  Plebe  Summer 
program  management.  Three  levels  of  management  were  selected  to  gain  a  thorough 
cross-sectional  view  of  performance  measurement  practices.  The  three  levels  and  related 
interviews  are  as  follows: 

•  Commandant  of  Midshipman 

•  Officer  In  Charge  of  Plebe  Summer  Detail 

•  Assistant  Officer  In  Charge  of  Summer  Detail 

•  Company  Officers  (3) 

Prior  to  conducting  interviews,  interviewees  were  provided  a  Pre- interview 
package.  The  package  included  a  coversheet  that  introduced  the  purpose  and  intentions 
of  the  interview.  Moreover,  it  provided  the  interviewees  definitions  of  the  performance 
measurement  terms  associated  with  this  research.  The  purpose  of  providing  this  package 
to  interviewees  was  to  insure  similar  knowledge  and  understanding  of  the  escarch 
model.  The  pre- interview  package  is  shown  in  Appendix  C. 
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A  Plebe  Summer  Outcome  Measurement  Matrix  was  also  included  in  the  pre¬ 
interview  package.  This  matrix  provided  interviewees  a  summary  of  the  performance 
measurement  practices  defined  in  the  Naval  Academy’s  instructions  governing  the  Plebe 
Summer  program.  The  matrix  is  designed  using  the  steps  specified  in  Hatry  and 
Kopczynski’s  (1997)  Guide  to  Program  Outcome  Measurement.  Interviewees  are  asked 
to  review  the  matrix  while  considering  the  following  questions: 

•  In  your  opinion,  does  this  list  of  objectives  cover  all  the  objectives  of  Plebe 
Summer?  If  not,  please  list  additional  objectives. 

•  In  your  opinion,  what  are  the  outcomes  these  objectives  intended  to  achieve? 

•  What  are  the  indicators  of  each  outcome? 

•  How  is  data  on  each  indicator  collected? 

•  What  is  the  benchmark  for  each  indicator?  What  are  the  minimum  acceptable 
levels  of  performance  associated  with  each  indicator? 

The  interviews  were  semi- structured  and  used  open  ended  questions  to  give 
interviewees  latitude  in  their  answers.  Questions  were  general  and  covered  the  following 
topics  and  theme: 

•  What,  in  your  opinion,  is  the  primary  purpose  of  the  Plebe  Summer  Program? 

•  What,  in  your  opinion,  are  the  goals  and  objectives  of  Plebe  Summer? 

•  What  outcomes  are  associated  with  each  objective? 

•  How  do  you  determine  the  success  or  failure  of  Plebe  Summer  in  meeting  its 
goals  and  objectives? 

•  What  information  do  you  use  in  determining  the  success  of  failure  of  Plebe 
Summer? 

•  Using  the  provided  matrix  please  identify  what,  in  your  opinion,  are  the 
indicators  of  each  outcome  you  believe  are  used  to  measure  the  success  or 
failure  of  each  goal  or  objective? 
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•  What  data  do  you  collect  and  use  for  each  outcome  indicator? 

•  How  is  outcome  data  recorded  and  reported? 

•  What  benchmarks  do  you  use  in  order  to  evaluate  the  data  collected? 

E.  OUTCOME  MEASUREMENT  EVALUATION 

The  final  research  area  uses  Hatry  and  Kopczynski’s  (1997)  Guide  to  Program 
Outcome  Measurement  to  evaluate  performance  measurement  and  management  applied 
to  Plebe  Summer.  The  guide  provides  a  mechanism  to  assess  the  effectiveness  of  the 
current  outcome  measurement  process  applied  to  the  Plebe  Summer  program. 

Information  gathered  from  literature  reviews  and  expert  interviews  are  fused  with 
the  guide  to  provide  a  foundation  for  assessment.  Specific  Plebe  Summer  objectives, 
outcomes,  outcome  indicators,  and  breakouts  listed  in  governing  instructions  and  expert 
interviews  are  assessed  using  the  guide.  The  purpose  of  the  qualitative  assessment  is  to 
determine  the  effectiveness  of  the  Plebe  Summer  performance  measurement  and  outcome 
management  procedures. 

F.  CHAPTER  SUMMARY 

This  chapter  described  the  methodology  for  this  research.  The  research 
methodology  consists  of  literature  reviews,  expert  interviews,  and  analysis.  First,  a 
review  of  the  current  performance  measurement  and  outcome  management  literature  is 
conducted.  Next  a  thorough  review  of  Naval  Academy  instructions  governing  Plebe 
Summer  follows.  Interviews  of  six  Plebe  Summer  experts  are  conducted  to  identify  the 
performance  measurement  procedures  used  by  personnel  responsible  for  the  planning  and 
conduct  of  Plebe  Summer.  The  following  chapters  will  analyze  the  Plebe  Summer 
performance  measurement  and  outcome  management  procedures  then  provide 
conclusions,  recommendations  to  the  Naval  Academy,  and  recommendations  for  further 
research. 
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V.  DATA  ANALYSIS 


A.  INTRODUCTION 

This  chapter  analyzes  the  data  collected  from  the  review  of  Naval  Academy 
instructions  and  interviews  of  Plebe  Summer  experts.  Each  Plebe  Summer  objective  is 
examined  to  evaluate  the  performance  measurement  and  outcome  management  practices 
applied  to  Plebe  Summer  by  the  Naval  Academy. 

Developing  program  outcome  information  is  a  critical  step  in  developing  a 
high  quality  performance  measurement  system  for  programs  and  projects 
in  the  federal  government.  Good  program  management  requires  collection 
and  use  of  outcome  data  to  provide  guidance  for  improvement.  Without 
information  on  results,  managers  can  only  supervise  “inputs”  or  monitor 
processes.  Decisions  on  whether  the  program  is  actually  working  well  - 
or  what  need  changing  -  are  made  in  the  absence  of  hard  data  on  actual 
outcomes  (Hatry  and  Kopczynski,  1997,  p.l). 

This  analysis  determines  whether  Naval  Academy  employs  performance  measurement 

and  outcome  management  practices  effectively  to  achieve  Plebe  Summer  objectives. 

The  Guide  to  Program  Outcome  Measurement  consists  of  nine  process  and 
implementation  steps  and  is  intended  to  be  a  tool  used  by  managers  and  their  staffs  in 
building  reasonably  valid  and  reliable  outcome  monitoring  procedures.  Evaluating  the 
outcome  management  procedures  applied  to  Plebe  Summer  divides  these  nine  steps  into 
two  areas.  First,  this  evaluation  begins  by  identifying  the  objectives  of  Plebe  Summer. 
This  establishes  a  foundation  to  answer  the  question,  “What  is  successful  performance  of 
Plebe  Summer?”  (Hatry  and  Kopczynski,  1997,  p.l) 

The  second  step  is  to  evaluate  the  current  Naval  Academy  performance 
measurement  and  outcome  management  procedures  for  each  identified  objective.  Plebe 
Summer  objectives  are  subsequently  evaluated  independently  using  the  following 
criteria: 

•  Outcomes  (O).  Are  specific  outcomes  for  each  objective  identified?  Do 
outcomes  cover  each  element  of  the  objective  statement? 
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•  Outcome  Indicators  (01).  Are  outcome  indicators  that  specifically  identify 
what  is  to  be  measured  identified?  Do  indicators  1)  measure  an  important 
aspect  of  the  outcome,  2)  overlap  with  other  indicators,  3)  cover  all  outcomes, 
and  4)  provide  relevant  feedback?  Are  indicators  controllable  by  the 
program? 

•  Data  collection  (DC).  Are  reasonable  data  sources  available?  Are  data 
collection  procedures  for  each  indicator  identified?  Does  data  collected  allow 
for  comparison? 

•  Benchmarks  (BM).  Are  benchmarks  that  provide  the  ability  to  assess  the 
successful  accomplishment  of  each  objective  identified? 

To  be  characterized  as  effective  the  performance  measurement  procedures  and  outcome 
management  procedures  of  each  objective  must  meet  these  criteria. 

B.  PLEBE  SUMMER  OBJECTIVES 

The  first  step  in  developing  an  outcome  monitoring  process  is  to  prepare  a 
mission/objectives  statement  for  the  program.  This  identifies  what  overarching  results 
are  hoped  to  be  achieved  or  the  purpose  toward  which  the  program  is  directed  (Hatry  and 
Kopczynski,  1997,  p.17).  Identifying  program  objectives  is  the  starting  point  for 
identifying  outcomes  and  performance  to  be  measured  later. 

The  basic  form  an  objective  statement  should  include  both  a  “To”  statement  and  a 
“By”  statement.  The  “To”  statement  identifies  the  basic  objectives  the  program  seeks. 
The  “By”  statement  identifies  the  method  by  which  the  objectives  are  accomplished 
(Hatry  and  Kopczynski,  1997,  p.  1 8). 

Plebe  Summer  is  managed  using  the  following  military  instruction:  Fourth  Class 
Indoctrination  (Plebe  Summer)  COMDTMIDINST  1510.2B  (USNA,  2001b).  This 
instruction  lists  three  Plebe  Summer  objectives: 

•  Effect  the  transition  from  civilian  or  enlisted  to  officer  candidate  status. 
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•  Indoctrinate  and  train  midshipmen  in  proper  military  discipline,  courtesy, 
requirements  and  traits  of  special  trust  and  confidence,  teamwork,  and 
personal  accountability. 

•  Provide  basic  training  in  seamanship,  marksmanship,  military  drill,  watch 
standing,  and  customs  and  traditions  of  the  Naval  service  (USNA,  2001b,  p. 
6). 

Expert  interviews  identified  one  additional  Plebe  Summer  objective  not  identified 
in  Naval  Academy  instructions: 

•  Prepare  the  Plebe  class  for  integration/incorporation  into  the  Brigade  of 
Midshipmen  and  the  challenge  of  Plebe  academic  year. 

Although  this  last  objective  is  not  addressed  by  Naval  Academy  instruction  it  was 
a  universal  theme  among  interviewees.  Commander  Patricia  Cole,  USN  expressed  this 
objective  saying: 

To  prepare  the  incoming  class  with  the  rudimentary  knowledge  and  skills 
so  that  they  are  equipped  and  prepared  to  be  incorporated  into  the  Brigade 
of  Midshipmen.  They  are  in  an  indoctrination  process  that  goes 
throughout  the  entire  year  and  this  is  just  a  part  of  it.  Plebe  summer  is  to 
get  them  into  physical  conditioning  and  teach  them  the  basic  military 
skills  and  knowledge  about  the  Naval  Academy  and  their  responsibilities 
and  expectations.  (Personal  Communication,  April  2002) 

The  Commandant  of  Midshipmen,  Colonel  John  Allen,  USMC  amplified  this  idea 

saying: 


In  a  very  real  sense  the  focused  purpose  of  Plebe  Summer  is,  through  an 
environment  of  imposed  discipline,  to  transition  them  from  individual 
priorities  to  collective  and  small  unit  priorities  associated  with  institutional 
excellence  as  a  member  of  the  fourth- class  regiment.  And,  then  have  them 
transition  out  of  the  fourth-class  regiment  and  into  the  Brigade  of 
Midshipmen.  (Personal  Communication,  April  2002) 

These  four  objectives  provide  adequate  “To”  statements.  However,  the  Naval 
Academy  instructions  governing  Plebe  Summer  do  not  contain  a  “By”  statement.  The 
instruction  that  governs  Plebe  Summer  provides  limited  clarity  on  how  objectives  are  to 
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be  met  when  it  defines  the  scope  of  Plebe  Summer.  This  definition  provides  nebulous 
direction  to  the  methods  employed  by  Plebe  Summer: 


Plebe  training  encompasses  progressive,  demanding  training  in  those 
subjects  required  to  produce  midshipmen  who  function  effectively  in  the 
brigade,  are  trained  in  drill,  are  physically  fit,  and  practice  those  personal 
and  professional  traits  which  distinguish  them  as  midshipmen  at  the 
United  States  Naval  Academy  (2001b,  p.  6). 

This  scope  declaration  is  vague  and  incomplete.  It  does  not  define  the  terms, 
“progressive”  and  “demanding”  or  define  what  “personal  traits”  distinguish  plebes  as 
midshipmen.  The  “By”  statement  should  lead  management  personnel  to  exactly  what 
should  be  measured  by  the  performance  measurement  system.  However,  in  the  case  of 
Plebe  Summer,  no  clear  guidance  is  provided.  The  absence  of  a  clear  and  direct 
statement  associated  with  each  objective  leads  to  misinterpretation  of  the  objectives  of 
Plebe  Summer.  This  misinterpretation  was  expressed  by  Naval  Academy  experts  when 
they  discussed  the  objectives  of  Plebe  Summer: 

•  There  are  all  kinds  of  factors  that  I’m  not  sure  we,  as  an  institution, 
have  properly  or  accurately  articulated  (Colonel  John  Allen, 

USMC,  Personal  Communication,  April  2002) 

•  There  is  a  sense  that  plebe  summer  without  a  high  attrition  rate 
wasn’t  really  a  Plebe  Summer.  There  is  the  sense  that  Plebe 
Summer  is  a  stand-alone  event,  an  experience.  None  of  that  is 
right.  There  will  be  some  attrition  during  Plebe  Summer,  but  the 
intent  of  Plebe  Summer  is  not  attrition.  (Colonel  Allen,  Personal 
Communication,  April  2002) 

•  During  Plebe  Summer,  I  think  there  is  a  misconception  by  some  of 
the  detailers  that  they  are  preparing  the  plebes  for  battle,  preparing 
them  for  war,  or  trying  to  train  them  to  be  these  people  who  finish 
plebe  summer  and  get  immediately  flown  over  to  combat 
terrorism.  Which  isn’t  the  case.  (Captain  Tyrel  Moxey,  USMC, 
Personal  Communication,  April  2002) 

A  complete  mission/objective  statement  is  important  in  developing  a  valuable 
performance  measurement  and  outcome  management  system.  It  is  the  starting  point  for 
identifying  the  outcomes  to  be  measured  and  the  specific  performance  indicators  that  are 
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needed  (Hatry  and  Kopczynski,  1997,  p.  17).  The  absence  of  a  “By”  statements  for  these 
four  objectives  forces  Academy  personnel  to  develop  ad  hoc  performance  indicators. 

Notwithstanding  the  lack  of  a  “By”  statement  these  four  objectives  provide  a 
foundation  for  analyzing  Plebe  Summer  performance  measurement  and  outcome 
management  procedures.  Each  address  one  aspect  of  the  purpose  toward  which  Plebe 
Summer  is  directed.  Although  there  is  no  “By”  statement  associated  with  the  objectives, 
the  objectives  do  expose  the  ends  toward  which  Plebe  Summer  is  designed  to  achieve. 

C.  OUTCOME  ANALYSIS 

The  remainder  of  this  chapter  evaluates  the  performance  measurement  and 
outcome  management  procedures  for  each  of  the  above  four  objectives.  The  following 
criteria  are  used  in  assessing  the  Plebe  Summer  performance  measurement  procedures 
identified  during  the  literature  review  and  expert  interviews:  outcome  identification, 
outcome  indicators,  data  collection  procedures,  and  benchmarks.  The  outcome  and 
performance  measurement  procedures  should  assist  Plebe  Summer  leaders  in  focusing  on 
results,  not  just  work  activities. 

There  is  a  hierarchical  pattern  to  the  four  listed  objectives.  The  first  objective  is 
focused  on  the  cultural  indoctrination  of  plebes.  The  second  objective  is  focused  on 
furnishing  plebes  with  the  tools  to  be  successful  within  the  Academy’s  unique  culture. 
The  third  objective  is  intended  to  provide  plebes  with  necessary  skills.  Finally,  the  last 
objective  is  intended  to  insure  plebes  are  prepared  apply  these  tools  and  skills  as  they 
enter  the  Brigade  of  Midshipmen.  Thus,  the  hierarchy  of  objectives  applied  to  Plebe 
Summer  can  be  listed  as  culture,  tools,  skills,  and  application. 

1.  Civilian  To  Enlisted  Transition 

The  first  objective  evaluated  is  the  transition  from  civilian  and  enlisted  to  officer 
candidate  status.  This  objective  is  focused  on  immersing  plebes  in  the  Academy’s  culture 
and  inculcating  plebes  with  the  cultural  norms.  Table  1,  the  Civilian  to  Officer  Candidate 
Transition  worksheet  displays  the  results  of  the  analysis.  Along  the  top  criteria  categories 
are  listed.  Then,  each  row  defines  an  outcome,  lists  outcome  indicators,  states  whether 
the  criteria  are  met,  or  indicates  whether  the  measurement  procedures  for  that  outcome 
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indicator  are  effective.  First  the  outcomes  are  evaluated;  then  each  criterion  is  evaluated 
and  explained  in  detail. 

a.  Outcome  Identification 

This  objective  is  very  weakly  defined.  Although  the  objective  seems  clear 
and  concise  at  first  glance,  it  contains  no  definition  of  “officer  candidate  status”.  Naval 
Academy  experts  agree  that,  to  them,  this  objective  is  linked  to  a  cultural  indoctrination 
process  that  occurs  during  Plebe  Summer.  Captain  Paul  Funk,  USMC,  described  the 
focus  of  this  cultural  indoctrination  process: 

To  instill  in  this  incoming  plebe,  this  civilian,  a  heightened  sense  of  and 
awareness  to  duty.  It’s  hard  to  quantify.  To  install  a  shared  set  of  values 
among  the  plebes  that  are  embraced  by  the  brigade.  Duty,  honor,  courage, 
commitment,  ship- shipmate- self.  To  guide  that  transition  and  it’s  changes. 

Then  you  see  the  plebes  actually  embrace  them  and  understand  them. 
(Personal  communication,  April  2002) 

Lieutenant  Colonel  Joseph  Dauplaise,  USMC,  Plebe  Summer  Officer- in- Charge,  supports 
this  statement: 

We  are  taking  a  raw  recruit  right  out  of  high  school  that  has  an  academic 
and  physical  background  and  no  known  character  flaws  and  bring  them 
into  this  culture.  We  now  have  to  transition  them.  We  have  to  break  them 
into  a  routine  where  you  get  up  early  and  stay  up  late.  So,  we  are  breaking 
the  civilian  out  of  them  and  putting  the  military  spin  on  them.  (Personal 
communication,  April  2002) 

There  is  no  Naval  Academy  instruction  that  defines  a  specific  outcome  to 
be  measured  for  this  objective.  However,  academy  experts  reported  that  an  outcome  (O) 
associated  with  this  objective  as:  increased  military  performance  during  Plebe  Summer. 
This  outcome  includes  professional  military,  physical,  and  academic  aspects  of  plebe 
performance. 

This  outcome  successfully  encompasses  the  entire  cultural  basis  of 
midshipmen  by  focusing  on  all  three  aspects:  professional  military,  physical,  and 
academic  performance.  Midshipmen  are  college  students  pursuing  an  academic  degree 
while  simultaneously  living  within  a  military  culture  that  includes  values,  dress  code, 
military  courtesy,  vocabulary,  and  personal  discipline.  The  combination  of  these  aspects 
creates  conditions  somewhat  uncommon  among  other  educational  institutions  and  is 
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important  when  addressing  the  cultural  indoctrination  of  plebes.  In  the  absence  of 
direction  provided  by  governing  instruction,  “increased  military  performance  during 
Plebe  Summer”  is  a  suitable  outcome  for  analyzing  this  objective. 


Table  1.  Civilian  to  Officer  Candidate  Transition  Worksheet 


Outcome 

OIs 

Data  Collection 

Benchmark 

Effective/ 

Procedures 

Non  Effective 

1 .  Increased  Military 

1 .  Military  Performance 

1.  Y 

1 .  N  (Individual  only) 

1.  NE 

Performance  during 
Plebe  Summer 

2.  Rate  Knowledge 

2.  Y 

2.  N  (Individual  only) 

2.  NE 

3.  PRT  Scores 

3.  Y 

3.  N  (Individual  only) 

3.  NE 

4.  Time  MOT  Skill 

4.  N 

4.  None 

4.  NE 

5.  Conduct/Honor 

5.  Y 

5.  None 

5.  NE 

b.  Outcome  Indicators 

Specific  outcome  indicators  (OIs)  for  this  objective  that  support 
performance  measurement  of  Plebe  Summer  are  not  listed  by  Naval  Academy 
instructions.  However,  Naval  Academy  experts  list  OIs  they  apply  to  this  outcome: 

•  Percent  receiving  passing  military  fitness  report  grades  during  Plebe 
Summer 

•  Percentage  passing  Physical  Readiness  Test  (PRT)  during  Plebe 
Summer 

•  Percentage  passing  rate  knowledge  quizzes  during  Plebe  Summer 

•  Incidence  of  demonstrated  poor  time  management  skills  during  Plebe 
Summer 

•  Incidence  of  conduct  or  honor  infractions  during  Plebe  Summer 

•  Rate  of  attrition  during  Plebe  Summer 

Each  of  these  indicators  measures  important  aspects  of  the  outcome. 
Fitness  report  grades  provide  an  accurate  description  of  midshipmen  performance 
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(USNA,  2001b,  p.  5-25).  Rate  knowledge  quizzes  are  designed  to  measure  plebe 
understanding  of  “their  individual  responsibility  to  further  his/her  training  and 
professional  development  and  to  facilitate  the  smooth  and  efficient  operation  of  the 
organization  itself’  (USNA,  2001b,  p.  2-3).  Moreover,  fitrep  grades  directly  represent 
the  level  to  which  individual  plebes  perform  professionally.  PRT  scores  are  important 
due  to  the  physical  requirements  placed  on  midshipmen.  All  midshipmen  are  subjected 
to  a  demanding  schedule  that  requires  excellent  time  management  skills.  Midshipmen 
are  expected  to  act  and  conduct  themselves  as  public  figures.  Therefore,  incidence  of 
honor  and  conduct  offenses  are  used  to  indicate  the  level  to  which  midshipmen  meet  this 
requirement.  Finally,  the  attrition  rate  is  keenly  scmtinized  because  it  indicates  the  level 
of  plebe  success  in  making  the  transition  into  the  Academy’s  culture. 

These  OIs  overlap  in  some  instances.  Due  to  the  demanding  schedules  to 
which  midshipmen  are  subjected  rating  and  professional  grades  can  be,  in  some 
instances,  due  to  poor  time  management  skills.  Attrition  can  be  affected  by  conduct  or 
honor  offenses. 

Each  indicator  provides  relevant  feedback  to  Plebes  and  Naval  Academy 
leaders  on  the  success  of  meeting  this  objective.  More  importantly,  most  of  these 
indicators  are  very  easy  to  understand  and  apply.  Problems  developing  time  management 
skills  are  more  difficult  to  identify  due  to  many  ways  poor  time  management  manifests 
itself.  However,  when  time  management  is  identified  as  a  source  of  difficulty  for  the 
plebe,  the  resultant  feedback  is  extremely  relevant  and  useful.  Attrition  rates 
significantly  higher  than  average  indicates,  to  Academy  leadership,  that  the  plebe  class  is 
having  difficulty  adjusting. 

Programs  should  have  some  control  over  outcome  indicators  for  the 
indicators  to  be  valid.  However,  this  criterion  should  not  be  over-valued.  As  long  as  the 
program  is  expected  to  ultimately  have  some  tangible  measurable  effect  on  the  outcome, 
it  should  be  included  (Hatry  and  Kopczynski,  1997,  p.31).  Plebe  Summer  is  successful  in 
providing  some  measure  of  control  over  each  indicator.  Plebe  Summer  experts  agreed 
that  successful  introduction  to  academy  culture  results  in  higher  performance  grades  both 
academic  and  military.  Also,  PRT  grades  rise  due  to  the  demanding  physical 
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requirements  of  the  Plebe  Summer  curriculum.  Additionally,  Plebe  Summer  experts 
maintain  that  time  management,  honor,  and  conduct  problems  decrease  as  plebe  s  accept 
the  tenets  of  personal  discipline  that  are  also  an  integral  part  of  the  Plebe  Summer 
curriculum  (Capt  Tyrel  Moxey  USMC,  Personal  Communication,  April  2002;  Colonel 
John  Allen,  USMC,  Personal  Communication,  April  2002;  Lieutenant  Joseph  Deauplais, 
USMC,  Personal  Communication,  April  2002). 

c.  Data  Collection  Methods 

Plebe  Summer  instructions  do  not  stipulate  specific  data  collection 
procedures  in  support  of  performance  measurement.  However,  data  is  collected  for  each 
OI  with  the  exception  of  time  management  skills.  Again,  time  management  skills  are 
difficult  to  quantify  and  measure  due  to  the  many  ways  poor  time  management  manifests 
itself. 

Fitrep  grades  are  assigned  by  the  squad  leaders  twice  during  the  summer. 
They  are  then  approved  by  the  Company  Officer  (USNA,  2001b,  p.  5-25).  However,  no 
comprehensive  procedures  are  defined  that  would  allow  class- wide  collection  of  this  data. 
PRT  scores,  conduct/honor  issues,  and  attrition  rates  are  reported  and  recorded  into  the 
Midshipman  Information  Data  System.  This  database  is  easily  accessible  to  every 
academy  leader.  Moreover,  the  information  is  consistently  recorded  every  academic 
semester.  Therefore,  there  is  no  missing  data  when  evaluating  the  entire  class  or 
individuals  within  the  class. 

Plebe  Summer  grades,  including  fitrep  information,  inspection  results  and 
rate  knowledge  are  established  and  recorded  by  midshipmen  squad  leaders.  Squad 
leaders  record  both  positive  and  negative  aspects  of  performance  in  their  Squad  Leader’s 
Notebooks  using  the  Plebe  Evaluation  Record  sheet.  The  purpose  of  this  record  sheet  is 
to  record  any  item  considered  significant  in  the  plebe’ s  development.  Items  recorded 
include  personal  appearance,  military  bearing,  attitude,  professional  knowledge,  self- 
confidence,  physical  performance,  watch  standing,  drill  performance,  uniform  and  room 
appearance.  Criteria  are  graded  by  the  squad  leader  as  below  average,  average,  or  above 
average.  The  importance  of  squad  leader  data  collection  was  articulated  by  Captain 
Moxey,  USMC: 
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We  made  sure  that  the  squad  leaders  were  writing  everything  down.  Each 
squad  leader  had  a  binder  so  there  was  a  binder  for  all  the  plebes  and  the 
record  of  counseling  were  put  in  respective  plebe’s  binder.  Whatever  it 
may  be,  good,  bad,  or  indifferent  counseling.  The  counseling  that  needed 
to  be  done  on  a  periodic  basis  was  put  in  there.  On  the  spot  counseling 
that  was  both  positive  and  negative  was  put  in  there.  Anything.  We 
highly  encouraged  them  to  put  it  down.  Because  we  continually  impress 
upon  them,  “if  you  don’t  write  it  down  it  didn’t  happen.”  You  can  say, 

“This  person  continually  can’t  do  whatever,  but  if  you  don’t  write  it  down 
nobody  knows  it  and  it’s  gone.” 

So,  we  collected  that  information  so  that  when  we  went  through  and 
basically  had  to  say,  “these  plebes  get  A’s  and  these  plebes  get  B’s,  and  so 
on”  we  had  information  to  go  on.  Now,  was  it  hard  numbers?  Did  we 
have  a  point  scale?  No.  It  was  this  person  has  three  or  four  great 
counseling  on  demeanor  and  discipline  and  punctuality.  And  this  one  here 
has  been  constantly  corrected  for  dirty  room,  messed  up  uniform,  there’s 
lots  of  documentation  here,  so  who  gets  the  A  and  whatever.  It  becomes 
pretty  obvious  (Personal  Communication,  April  2002). 

The  information  recorded  by  squad  leaders  is  not  shared  class- wide. 
Therefore,  there  is  no  method  to  assemble  information  concerning  entire  class 
performance  into  a  single  database  that  affords  comparison  class-wide.  Moreover, 
comparison  against  previous  performance  or  pre- determined  targets  is  impossible. 

d.  Benchmarks 

No  minimum  standards  or  targets  are  assigned  on  a  program-wide  basis. 
Fitrep  grades  have  minimum  passing  standards  on  an  individual  basis  based  on  an  A  thru 
F  (4.0)  scale  with  minimum  “C”  grade  for  passing.  They  indicate  individual  performance 
and  comparison  against  previous  classes  or  predetermined  targets  is  impossible. 

PRT  grades  are  also  based  on  an  A  thru  F  (4.0)  scale  with  minimum  “C” 
grade  for  passing.  PRT  grades  are  established  using  a  300-point  system  with  minimum 
performance  levels  defined  for  each  of  the  three  physical  tests  (pushups,  sit- ups,  and  1.5 
mile  run)  each  worth  100  points.  These  grades  are  easily  summarized  and  can  be  used  to 
compare  with  previous  plebe  classes,  pre-selected  targets,  or  different  program  practices 
(such  as  differing  physical  development  techniques).  However,  no  minimum  class- wide 
benchmarks  are  set  down  for  comparison. 
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Attrition  rates  also  have  no  minimum  or  maximum  targets.  However, 
occurrence  rates  can  easily  be  summarized  and  compared  with  previous  plebe  classes. 
The  same  applies  for  the  incidence  of  conduct  and  honor  offenses.  Although  no 
minimum  standard  is  set,  the  data  collected  can  also  be  compared  to  previous  classes.  All 
conduct  and  honor  offenses  are  adjudicated  and  remediated  within  a  short  period  after 
infraction  on  a  case-by-case  basis  according  to  military  guidelines.  If  an  explanation  or 
training  deficiency  is  identified  as  a  contributing  factor  to  the  offense,  remediation  is 
arranged  to  assist  midshipman  development.  There  are  no  benchmark  criteria  for 
measuring  incidence  of  poor  time  management  skills. 

e.  Conclusion 

The  performance  measurement  and  outcome  management  procedures 
associated  with  the  civilian  or  enlisted  to  officer  candidate  transition  are  not  effective. 
There  are  no  specific  performance  measurement  procedures  defined  through  Plebe 
Summer  instruction.  However,  academy  experts  detailed  an  outcome  as  well  as  a  number 
of  outcome  indicators  and  data  collection  procedures  they  use  in  ad  hoc  fashion  to 
measure  the  success  of  this  outcome. 

The  absence  of  a  definition  of  “officer  candidate  status”  limits  Academy 
personnel’s  ability  to  accurately  measure  the  success  of  this  objective.  Until  definitions 
are  identified  and  specific  procedures  are  established  academy  personnel  will  continue  to 
use  ad  hoc  measures  of  performance. 

Although  individual  performance  benchmarks  are  identified  for  each 
outcome  indicator  there  are  no  minimum  benchmarks  to  compare  the  class- wide  success. 
Therefore,  Plebe  Summer  leadership  cannot  accurately  assess  the  level  of  success  the 
program  has  in  effecting  the  civilian  to  officer  candidate  transition. 

2.  Indoctrinate  and  Train  Midshipmen 

The  second  objective  is  intended  to  provide  plebes  with  a  number  of  tools  they 
will  employ  as  midshipmen  within  the  academy’s  culture.  The  objective  states  five  areas 
that  midshipmen  receive  indoctrination  and  training:  1)  military  discipline,  2)  military 
courtesy,  3)  requirements  and  traits  of  special  trust  and  confidence,  4)  teamwork,  and  5) 
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personal  accountability.  This  objective  builds  upon  the  first  in  that  it  is  intended  to 
provide  plebes  with  the  tools  to  effectively  survive  in  the  Academy’s  military  culture. 
Table  2,  Plebe  Indoctrination  and  Training  Worksheet  displays  the  results  of  analysis. 
a.  Outcome  Identification 

There  is  no  specific  outcome  established  for  this  objective.  Moreover,  the 
objective  is  poorly  worded  and  ambiguous.  The  terms  included  in  this  objective 
(discipline,  courtesy,  team  work,  special  trust  and  confidence)  are  not  defined.  This 
forces  Plebe  Summer  experts  to  develop  ad  hoc  definitions  and  performance  metrics  in 
order  to  measure  the  success  of  this  objective.  The  Plebe  Summer  experts  identify  that 
the  desired  outcome  of  Plebe  Summer  associated  with  this  objective  is:  plebes  learn  and 
demonstrate  military  professionalism  and  personal  discipline. 

Table  2.  Plebe  Indoctrination  and  Training  Worksheet 

OIs  Data  Collection  Benchmark 

Procedures 

1 .  Conduct/Honor  1 .  Y  1 .  N  (Individual  only) 

2.  Personal  Observations  2.  Y  2.  N  (Individual  only) 

3.  Inspection  Results  3.  Y  3.  N  (Individual  only) 

4.  Rate  Quiz  results  4.  N  4.  None 


This  outcome  focuses  exclusively  on  professional  military  aspects  of  plebe 
development  as  discussed  by  Commander  Patricia  Cole,  USN: 

What  I  would  look  for  is,  as  far  as  the  training,  do  they  seem  to  be 
handling  their  responsibilities  well?  Do  they  know  where  they  are 
supposed  to  be?  Do  they  know  how  to  correctly  wear  a  uniform?  Do  they 
know  how  to  address  people  who  are  senior  to  them?  Do  they  know  how 
to  use  the  chain- of- command?  Basic  military  things.  (Personal 
communication,  April  2002) 

Although  no  specific  outcome  is  listed  Plebe  Summer  instruction  this  objective  will  be 
analyzed  using  the  above  outcome  obtained  from  Academy  experts. 


Outcome 


1.  Learn  and 
Demonstrate  Military 
Professionalism  and 
Personal  Discipline 


Effective/ 
Non  Effective 

1.  NE 

2.  NE 

3.  NE 

4.  NE 
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b. 


Outcome  Indicators 


Plebe  Summer  experts  listed  various  indicators  associated  with  this 
outcome.  Included  in  the  list  are: 

•  Low  number  of  conduct  or  honor  infractions 

•  Incidence  of  outstanding  scores  on  uniform  and  room  inspections 

•  Incidence  of  professionalism  observed  by  senior  personnel 

•  Percentage  passing  rate  knowledge  quizzes  during  Plebe  Summer 

These  indicators  focus  specifically  on  military  performance  and  the  demonstration  of 
professional  military  ideals.  Although  Plebe  Summer  instructions  do  not  specify  these 
indicators  as  metrics  for  this  objective,  they  will  be  used  for  this  analysis. 

Each  01  measures  an  important  aspect  of  the  outcome.  Conduct  and 
honor  offenses  indicate  understanding  and  acceptance  of  the  military  culture  to  which 
plebes  are  being  introduced.  Moreover,  low  incidence  levels  of  conduct  and  honor 
infractions  indicate  high  levels  of  personal  discipline  among  the  plebes.  Room  and 
uniform  inspection  results  are  indicators  of  plebe  attention  to  detail  and  motivation 
towards  maintaining  high  personal  standards.  Demonstrated  professionalism  such  as 
proper  courtesy  toward  senior  personnel  indicates  comprehension  of  the  professional 
training  plebes  receive  as  well  as  personal  discipline.  Finally,  high  performance  on  rate 
quizzes  indicates  plebe  understanding  of  their  personal  responsibility  to  self- 
improvement  and  professional  education. 

There  is  overlap  present  among  these  OIs.  Conduct/honor  offenses  are 
incorporated  into  military  performance  grades.  Therefore,  these  indicators  are  duplicated 
when  measuring  the  outcome  of  this  objective.  Also,  senior  personnel  such  as  company 
officers  and  senior  enlisted  advisors  have  input  to  military  performance  grades.  In  this 
situation,  their  personal  reflections  are  recorded  twice. 

Each  indicator  listed  provides  relevant  feedback.  Furthermore,  they  are 
easy  to  understand.  High  rates  of  conduct  and  honor  offenses  quickly  indicate  problems 
with  plebe  professionalism  and  a  high  incident  rate  among  the  entire  class  could  indicate 


63 


poor  training  of  expectations.  Inspection  results,  rate  quiz  results,  and  personal 
observations  also  provide  the  same  quick  indication  of  plebe  professionalism  and 
motivation. 

Plebe  Summer  training  provides  plebes  with  training  and  tools  necessary 
to  conduct  themselves  professionally  within  the  academy  culture.  Although  Plebe 
Summer  training  cannot  control  conduct  or  honor  offenses  it  does  define  for  plebes 
acceptable  behavior  and  addresses  the  conduct  expectations  of  each  midshipmen. 
Therefore,  Plebe  Summer  does  maintain  some  control  over  these  outcome  indicators. 

c.  Data  Collection  Methods 

Plebe  military  performance  is  observed  and  recorded  by  their  squad  leader 
in  the  same  fashion  describe  in  the  first  objective  analysis.  Squad  leaders  record  both 
positive  and  negative  aspects  of  performance  using  the  Plebe  Evaluation  Record  sheet. 
Conduct  offenses  are  also  recoded  (USNA,  2001b,  p.  71).  These  records  are  used  as  a 
tool  when  squad  leaders  provide  performance  counseling  to  plebes.  However,  the  records 
of  individual  plebe  performance  are  not  collected  for  use  outside  the  company. 

Inspection  grades  and  rate  quiz  performance  are  also  determined  and 
recorded  by  squad  leaders.  Plebe  performance  on  inspections  and  rate  knowledge  quizzes 
is  recorded  by  squad  leaders  using  the  Plebe  Evaluation  Record  sheet.  These  reports  are 
not  consolidated  class-wide  using  a  centralized  database.  Therefore,  these  data  are  not 
easily  summarized  and  used  by  senior  personnel.  Moreover,  individual  and  company 
average  inspection  performance  is  not  recorded  or  tracked  outside  the  company  level. 

Incidence  of  professionalism  observed  by  senior  personnel  are  not 
recorded  or  tracked  by  anyone  other  than  that  person  making  the  observation.  No  central 
data  collection  or  recording  is  done  with  this  information. 

No  data  are  collected  on  “teamwork”.  Moreover,  Plebe  Summer  experts 
do  not  indicate  understanding  on  how  teamwork  could  be  measured: 

Training  “teamwork”?  I  don’t  know,  is  there  a  metric  for  that?  Is  there  a 
written  test?  There  wasn’t  for  my  company  and  I  don’t  know  if  you  could 
create  one  (Captain  Tyrel  Moxey,  USMC,  Personal  Communication,  April 
2002) 
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An  easy  performance  tool  is  knowledge  and  certain  other  standards  like 
PRT  scores.  The  harder  to  quantify  ones  are  performance  related  such  as 
“teamwork”.  Or,  a  person’s  bearing,  how  well  they  hold  themselves. 

How  well  they  react  to  detailers.  Heck,  it’s  even  hard  to  quantify  if 
somebody  understands  teamwork  or  not.  It’s  a  small  thing,  but  how  do 
you  look  at  someone  and  give  them  a  grade  on  teamwork?  You  can’t. 
(Captain  Paul  Funk,  USMC,  Personal  Communication,  April  2002) 

Although  teamwork  can  be  demonstrated  it  is  more  difficult  to  quantify  and  compare 

among  a  group. 

d.  Benchmarks 

Benchmarks  that  allow  data  comparisons  are  not  defined.  Plebe  summer 
leadership  use  improvised  methods  to  determine  success  in  meeting  this  objective. 
Plebes  are  ranked  using  the  squad  leader  evaluation  records;  however,  this  ranking  is  kept 
in-company,  and  no  comparison  among  other  companies  is  conducted. 

Fitrep  grades,  rate  knowledge  quiz  performance,  and  inspection  results 
have  no  minimum  allowable  standard.  Rather,  problems  are  identified  and  plebes  receive 
immediate  feedback  on  how  to  correct  deficiencies.  A  further  problem  with  inspections 
is  that  inspection  criteria  are  not  standardized  resulting  in  no  two  plebes  receiving  the 
same  level  of  scrutiny.  Conduct  and  honor  offenses  are  recorded  in  the  MIDS  database. 
This  database  provides  easy  access  to  Academy  leadership. 

e.  Conclusion 

The  performance  measurement  procedures  associated  with  this  objective 
deal  specifically  with  individual  performance  and  not  program  evaluation.  Therefore,  the 
performance  measurement  of  indoctrinating  and  training  plebes  in  military  discipline, 
military  courtesy,  requirements  and  traits  of  special  trust  and  confidence,  teamwork  and 
personal  accountability  is  not  effective.  Teamwork  data  are  not  collected  due  to  the 
complex  nature  of  the  subject  matter.  Moreover,  data  collection  methods  for  the  other 
OIs  do  not  gather  information  on  a  class-wide  basis  in  order  to  afford  comparison  with 
previous  performance  or  targets. 

Plebe  summer  personnel  could  not  effectively  measure  the  success  of  this 
objective  even  if  class-wide  data  was  available.  This  is  due  to  the  complete  lack  of 
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benchmarks  for  data  comparison.  Although  individual  performance  benchmarks  are 
identified,  they  do  not  assist  in  assessing  objective  success. 

3.  Provide  Basic  Training 

The  third  defined  objective  of  Plebe  Summer  is  to  provide  plebes  with  basic 
training  in  various  military  subjects  including:  seamanship,  marksmanship,  military  drill, 
watch  standing,  and  customs  and  traditions  of  the  Naval  Service.  This  objective  is 
designed  to  develop  within  plebes  a  skill  set  that  will  enhance  their  success  at  the  Naval 
Academy  and  in  the  fleet  as  Naval  officers. 

a.  Outcome  Identification 

No  outcome  for  this  objective  is  defined  through  Academy  instruction. 
The  Commandant  of  Midshipmen,  however,  articulates  the  intent  of  this  objective: 

Do  midshipmen  at  the  Naval  Academy  need  to  shoot  weapons?  The 
answer  is  no,  they  don’t  need  to  shoot  weapons.  But  do  we  produce  a 
better  midshipman  and  ultimately  a  better  officer  if  they’ve  had  the 
opportunity  to  go  through  the  process  of  learning  the  techniques  and 
procedures  associated  with  the  surgical  application  of  marksmanship? 

The  answer  to  that  question  is  yes.  Not  because  they  will  ever  have  to  fire 
that  weapon  again.  But  I  contend  that  an  officer  of  the  Naval  service, 
regardless  of  warfare  specialty  he  or  she  serves  in,  is  a  more  confident 
officer,  is  an  officer  who  feels  better  about  their  own  role  in  this 
profession  of  arms  if  they  have  some  sense  of  their  ability  to  handle 
weapons. 

In  conjunction  with  the  martial  arts  program  we’re  creating  this  gives 
them  a  sense  that  weapons  handling,  whether  you’re  standing  on  the 
bridge  of  a  ship,  staring  down  a  periscope,  flying  a  fighter,  or  leading  a 
rifle  platoon,  weapons  handling  is  part-in-parcel  with  the  profession  of 
arms. 

It’s  just  good  for  your  self-confidence.  So,  that’s  why  we  do  basic 
marksmanship.  Basic  drill,  of  course,  is  basic  teamwork  and  immediate 
obedience  to  orders.  Basic  drill,  close-order  drill  is  part- in-parcel  with  any 
early  regimentation  and  any  self-imposed  discipline.  (Personal 
communication,  April  2002) 

Plebe  Summer  experts  identified  the  outcome  they  associate  with  this 
objective  as:  plebes  are  equipped  with  the  skills  to  succeed  within  the  academy’s  culture. 
Plebes  draw  on  the  tools  acquired  in  previous  training  in  order  to  meet  professional 
challenges  of  watch  standing,  seamanship,  marksmanship,  and  military  drill.  Through 
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instruction  in  the  customs  and  traditions  of  the  Naval  service  plebes  learn  the  context  in 
which  they  apply  these  skills. 

Although  no  outcome  is  defined  through  Plebe  Summer  instruction  the 
outcome  identified  by  the  Commandant  of  Midshipman  will  be  used  for  analysis.  Table 
3,  Basic  Training  Worksheet,  displays  the  results  of  this  analysis. 


Table  3.  Basic  Training  Worksheet 


Outcome 

Outcome  Indicators 

Effective  Data 

Effective  Benchmarks 

Effective/ 

Collection 

Procedures 

Non  Effective 

1.  Plebes  are 

1.  Percent  passing 

1.  N 

1 .  N  (Individual  only) 

1.  NE 

Equipped  with  Skills 
Required  at  USNA 

academic  exams 

2.  Percent  Receiving 
sailing  qualification 

2.  N 

3.  N 

2.  N  (Individual  only) 

3.  N  (Individual  only) 

2.  NE 

3.  NE 

3.  Percent  receiving 
weapons  qualifications 

4.  N 

4.  N  (Company  only) 

4.  NE 

4.  Percent  of  Companies 
receiving  excellent  drill 

scores 

b.  Outcome  Indicators 

Outcome  indicators  used  as  metrics  for  this  outcome  measure  skill  level 
and  subject  understanding.  Indicators  currently  used  to  measure  success  of  this  outcome 
include: 

•  Percent  receiving  passing  marks  on  professional  instruction  in  Damage 
Control,  Visual  Signaling. 

•  Percent  earning  sailing  “B”  qualification 

•  Percent  receiving  expert  marksmanship  scores 

•  Percent  receiving  excellent  military  drill  scores 

The  indicators  measure  only  the  practical  application  and  understanding  of  instruction. 
The  information  obtained  through  these  indicators  does  not  provide  feedback  in  the 
context  of  naval  customs  and  tradition. 
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The  indicators  measure  important  aspects  of  the  desired  outcome.  The 
following  paragraphs  detail  the  professional  curriculum  as  well  as  the  curriculum 
objectives  described  in  this  objective.  Moreover,  each  subject  description  will  list  the 
skills  and  knowledge  measured  by  these  indicators  as  well  as  their  practical  application 
within  the  Academy’s  culture. 

Damage  control  is  a  basic  skill  of  all  sailors.  The  ability  to  fight  casualties 
onboard  a  ship  is  the  foremost  challenge  of  new  personnel.  Demonstrated  proficiency  in 
casualty  and  damage  control  is  a  hallmark  of  professional  naval  officers.  Through 
developing  expertise  in  damage  assessment  and  casualty  combat  plebes  develop  and  gain 
confidence  in  the  use  of  personal  decision-making  skills.  The  damage  control  curriculum 
plebes  receive  includes  fire-fighting  techniques,  flood  control,  damage  control  equipment 
and  its  applications.  Plebes  also  receive  hands-on  experience  plugging  and  patching 
piping,  donning  emergency  breathing  devices,  using  fire  extinguishers,  and  using  the 
Navy  all-purpose  nozzle  and  fire  hose  (USNA,  2001a,  p.35). 

Naval  ships  employ  visual  communications  regularly.  Midshipmen  are 
expected  to  attain  the  ability  to  encode  and  decode  visual  signals  rapidly.  This 
professional  skill  can  be  directly  applied  to  fleet  demands  and  achieving  visual 
communications  proficiency  provides  plebes  with  greater  understanding  of  the  demands 
placed  on  operational  naval  units.  The  visual  communications  curriculum  includes 
lessons  on  the  importance  and  use  of  visual  communications,  communications 
organization  format,  and  signaling  responsibilities.  Plebes  receive  hands-on  training  with 
Morse  code  and  Flag  hoist  signaling  (USNA,  2001a,  p.37). 

Ship  control  is  the  fundamental  skill  of  a  naval  officer.  Regardless  of 
warfare  specialty,  these  lessons  promote  the  development  of  skills  such  as  foresight, 
planning,  tenacity,  flexibility,  and  diligence.  It  matters  little  whether  plebes  eventually 
apply  these  lessons  to  ships,  planes,  or  riffle  platoons;  what  matters  is  the  development  of 
fundamental  skills  to  apply  in  an  environment  that  is  constantly  changing  to  produce 
varying  challenges.  Proficiency  in  this  subject  provides  plebes  with  confidence  in  their 
ability  to  work  within  a  constantly  changing  environment,  he  sailing  curriculum  includes 
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lessons  in  basic  seamanship,  sailing  procedures,  sailing  equipment,  emergencies,  and 
sailing  practical  (USNA,  2001a,  p.36). 


The  value  of  marksmanship  training  was  expressed  previously.  These 
lessons  provide  plebes  with  self-confidence  and  understanding  of  their  roles  within  a 
military  profession.  The  curriculum  includes  training  in  weapon  care  and  safety,  firing 
procedures,  and  live-fire  qualification.  Midshipmen  who  score  high  on  marksmanship 
testing  qualify  for  ribbons  worn  on  the  uniform.  There  are  three  levels  of  qualification 
for  both  the  M-16  rifle  and  the  9mm  pistol  are,  expert  (E),  sharpshooter  (S),  and  basic 
ribbon  qualification  (USNA,  2001a,  p.41). 

Military  drill  introduces  plebes  to  teamwork  and  self-discipline.  Lessons 
within  this  subject  enlighten  plebes  to  their  role  within  a  military  environment. 
Moreover,  military  drill  trains  midshipmen  in  the  skills  of  teamwork  and  obedience  to 
military  orders. 

The  OIs  for  this  objective  do  not  overlap  with  one  another  and  are 
controllable  by  the  Plebe  Summer  program.  Each  subject  is  tested  separately  and  the 
indicators  provide  relevant  feedback  to  both  Academy  leadership  and  plebes.  Successful 
completion  of  each  examination  and  practical  test  indicates  understanding  of  the 
curriculum  concepts  and  demonstrates  the  requisite  skills  required  for  later  plebe  success. 

c.  Data  Collection  Methods 

There  are  no  data  collection  procedures  for  this  objective.  Although  there 
are  grades  associated  with  the  Damage  Control,  Visual  Signals,  and  Sailing  exams,  the 
performance  of  individual  plebes  is  neither  recorded  nor  tracked  on  a  class- wide  basis.  In 
the  area  of  weapons  training,  midshipmen  who  qualify  for  a  ribbon  have  the  qualification 
noted  in  their  military  record.  However,  the  rate  of  qualification  among  the  plebe  class  is 
not  recorded.  The  same  is  true  for  sailing  “B”  qualification.  Each  plebe  successfully 
qualifying  has  it  noted  in  their  military  record.  However,  the  rate  of  qualification  among 
the  entire  class  is  not  recorded. 

Drill  performance  is  recorded  on  a  company  level  only  and  no  historical 
data  is  collected.  Therefore,  class  performance  cannot  be  compared.  Also,  drill  is  graded 
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on  a  company  wide  basis  only.  This  is  problematic  in  view  of  the  fact  individual  plebe 
performance  cannot  be  differentiated  from  the  company’s.  Poor  drill  performance  on  an 
individual  basis  is  not  recorded  resulting  in  difficulties  identifying  and  remediating  low 
performers. 

d.  Benchmarks 

There  are  no  benchmarks  defined  by  Naval  Academy  instruction  that 
allows  for  comparison  among  groups  or  previous  classes.  Plebe  summer  personnel  could 
not  effectively  measure  the  success  of  this  objective  even  class- wide  data  was  available. 
This  is  due  to  the  complete  lack  of  benchmarks  for  data  comparison.  Although  individual 
performance  benchmarks  are  identified,  they  do  not  assist  in  assessing  objective  success. 

e.  Conclusion 

The  performance  measurement  and  outcome  management  procedures  for 
this  objective  are  not  effective.  No  procedures  for  measuring  class- wide  performance  are 
detailed  by  Academy  instruction.  This  forces  academy  personnel  to  develop  individual 
performance  measures  to  apply  on  an  individual  basis. 

Although  the  desired  outcome  and  outcome  indicators  are  clearly  defined 
by  Plebe  Summer  experts,  they  are  unable  to  identify  clear  data  collection  methods  or 
benchmarks  that  adequately  conclude  the  success  or  failure  in  meeting  this  objective. 
Moreover,  data  collection  methods  and  benchmarks  used  by  Plebe  Summer  experts  all 
measure  individual  or  company- wide  performance  and  are  not  amassed  to  provide  class¬ 
wide  comparison  with  previous  performance  or  pre-selected  targets. 

4.  Prepare  Plebe  Class  For  Integration  Into  The  Brigade  of  Midshipmen 

The  last  objective  is  not  defined  by  Naval  Academy  instruction.  However,  it  was 
universally  identified  by  Plebe  Summer  experts  and  was  defined  as:  preparing  the  plebe 
class  for  integration/incorporation  into  the  Brigade  of  Midshipmen.  This  objective  ties 
the  previous  three  objectives  together  in  that  it  is  focused  on  teaching  the  plebe  class  how 
to  apply  the  previously  analyzed  tools  and  skills  within  the  Naval  Academy  and  military 
culture.  One  expert  described  this  objective: 
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I  would  say  the  things  that  plebes  learn  to  make  them  good  plebes  are  the 
same  things  that  will  make  them  good  midshipmen  and  good  naval 
officers.  They  are  not  separate.  The  aspects  of  punctuality,  being  able  to 
prioritize,  of  being  able  to  press  on  when  tired,  hungry,  and  worn  out.  All 
these  qualities  we  try  to  impart  during  plebe  summer  and  see  how  the  do 
with  the  struggles.  Some  rise  to  the  challenge,  others  do  not  perform  as 
well.  Nevertheless,  those  same  traits  that  exude  during  plebe  summer  are 
the  same  that  third-class  midshipmen,  second-class  midshipmen,  and 
firsties  are  all  tested  on  and  expect  in  plebes.  And  they  are  one  in  the 
same  to  those  that  make  a  great  naval  officer  (Personal  Communication, 
Capt  Tyrel  Moxey,  April  2002). 


Another  expert  described  this  objective  more  directly: 

Under  the  heightened  supervision  of  Plebe  Summer  and  heightened 
intensity  of  Plebe  Summer  we  take  plebes  and  turn  them  into  a  product 
that  we  can  place  into  the  academic  year  under  less  supervision  and  let  the 
brigade  finish  the  training.  Because  that  is  where  the  majority  of  training 
is  going  to  take  place  (Personal  Communication,  Capt  Paul  Funk,  April 
2002) 

The  Commandant  of  Midshipmen  underscores  the  importance  of  this  objective  and  the 

importance  of  the  transition  from  Plebe  Summer  to  the  academic  year: 

Now,  the  institution  here  has  not  done  a  very  good  job.  Has  not  done  a 
very  good  job  of  explaining  the  seam  ,  the  S-E-A-M,  between  Plebe 
Summer  and  the  formation  of  the  brigade.  As  well,  we  have  not  done  a 
very  good  job  as  an  institution  of  picking  up  the  momentum  of  the  plebe 
development,  the  fourth- class  development,  the  professional,  moral, 
mental,  and  physical  that  occurs  during  Plebe  Summer  and  have  that 
momentum  continue  in  the  academic  year  (Personal  Communication, 
Colonel  John  Allen,  April  2002). 


The  foundation  of  this  objective  is  that  the  professional  development  of  plebes 
does  not  stop  upon  completion  of  Plebe  Summer.  Rather,  it  continues  throughout  the 
fourth-class  academic  year  where  the  degree  of  supervision  and  control  place  upon  plebes 
is  greatly  reduced.  With  this  in  mind,  Plebe  Summer  must  inculcate  each  plebe  with 
methods  and  strategies  to  effectively  apply  the  tools  and  skills  learned  during  plebe 
summer  in  order  to  be  successful  during  the  academic  year. 
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a. 


Outcome  Identification  and  Analysis 


The  outcome  associated  with  this  objective  is:  plebe  development 
continues  during  the  academic  year.  This  outcome  is  based  on  the  proper  application  of 
the  tools  and  skills  learned  in  Plebe  Summer. 

I  typically  do  not  look  at  Plebe  Summer  in  isolation.  Because,  if  we’ve 
done  it  right,  Plebe  Summer  simply  prepares  the  fourth-classmen  to  join 
the  brigade  where  the  preponderance  of  training  continues.  Plebe  Summer 
simply,  through  the  mechanism  of  imposed  discipline  and  the  stressors 
and  strains  of  a  schedule  that  they  can  never  meet,  provides  them  the 
context  and  the  early  military  skills  and  qualities  necessary  to  function 
properly  within  the  brigade  (Personal  Communication,  Colonel  John 
Allen,  April  2002). 

At  first  glance,  this  outcome  seemingly  duplicates  the  outcome  describe 
for  the  first  objective.  This  last  objective  may  seem  as  though  it  is  an  extended 
enculturation  process.  However,  while  two  outcomes  a'c  closely  related  and  rely  on 
similar  outcome  indicators  and  benchmarks  there  are  distinct  differences  between  the 
two.  The  first  outcome  is  intended  to  improve  plebe  performance  as  a  function  of 
successful  enculturation  during  Plebe  Summer.  Moreover,  the  first  objective  measures 
success  within  the  timeline  of  Plebe  Summer.  However,  this  fourth  outcome  addresses 
continued  plebe  development  upon  the  conclusion  of  Plebe  Summer  as  a  function  of  the 
application  of  skills  and  tools  attained  during  plebe  summer.  Also,  the  outcome 
indicators,  data  collection,  and  benchmark  comparisons  associated  with  this  last  objective 
all  take  place  after  plebe  summer  concludes.  The  results  of  analysis  for  this  objective  is 
presented  in  Table  4,  Transition  to  Brigade  Preparation  Worksheet. 

b.  Outcome  Indicator  Analysis 

The  OIs  listed  for  this  objective  measure  the  professional  success  of 
midshipmen  during  the  academic  year. 

•  Percent  receiving  passing  academic  grades  and  overall  class  GPA 

•  Percent  receiving  passing  academic  year  military  performance  grades 
and  overall  MQPR 

•  Attrition  rate  during  the  academic  year 
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•  Incidence  of  conduct  or  honor  offences  during  academic  year 

•  Proportion  of  class  passing  PRT  during  academic  year 

These  OIs  each  measure  an  important  aspect  of  the  outcome.  Superior 
grades  in  academics,  professional  military  performance,  and  PRT  indicate  plebes 
competence  in  applying  time  management  and  other  skills  required  for  success  at  the 
academy.  Moreover,  military  performance  grades  measure  the  day-to-day  application  of 
the  professional  training  plebes  received  during  Plebe  Summer.  This  01  is  particularly 
important  because  it  measures  the  depth  of  professional  growth  during  the  academic  year. 
The  rate  of  honor  and  conduct  offenses  measures  plebe  ability  to  apply  the  concepts  of 
special  trust  and  confidence  as  well  as  personal  accountability  detailed  earlier  in  this 
research.  The  rate  of  attrition  measures  overall  capacity  of  the  plebe  class  in  adjusting  to 
and  meeting  the  continued  demands  after  academic  year. 

There  is  little  overlap  between  OIs.  Each  indicator  measures  a  unique 
aspect  of  the  desired  outcome.  However,  conduct  and  honor  offenses  make  up  a  small 
proportion  of  the  military  performance  grade.  Also,  conduct  and  honor  offenses  can  be 
the  root  cause  of  attrition. 

Relevant  feedback  is  provided  by  these  OIs.  Low  academic  or  military 
performance  grades  are  indicators  that  plebes  are  having  difficulty  meeting  the  day-to- 
day  challenges  of  the  Academy.  High  rates  of  conduct  or  honor  offenses  indicate  plebe 
inability  in  applying  aspects  of  personal  accountability  and  discipline.  Physical 
Readiness  Test  grades  indicate  plebe  motivation  in  maintaining  the  high  level  of  fitness 
acquired  during  Plebe  Summer. 

Although  the  curriculum  is  designed  to  provide  plebes  with  the  skills, 
tools,  as  well  as  methods  to  apply  them  within  the  Academy’s  culture  only  small  portions 
of  these  OIs  are  controllable  by  the  Plebe  Summer  curriculum.  Continued  plebe 
development  after  Plebe  Summer  is  dependent  on  other  outside  forces  such  as  the 
motivation  and  ability  of  Academy  personnel  in  this  continual  development. 
Furthermore,  individual  plebe  personal  situations  vary  widely.  The  effect  of  staff 
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motivation  and  training  or  the  plebe  personal  lives,  outside  the  academy,  are  not  taken 
into  consideration  with  these  indicators 

c.  Data  Collection  Method  Analysis 

No  data  collection  procedures  are  specifically  designed  to  support  Plebe 
Summer  performance  measurement.  However,  useful  data  is  collected  for  each  indicator 
that  allows  for  comparison  among  the  entire  plebe  class  and  between  plebe  years. 
Indicator  data  is  collected  and  stored  in  the  MIDS  database 


Table  4.  Transition  to  Brigade  Preparation  Worksheet 


Outcome 

Outcome  Indicators 

Effective  Data 
Collection 

Effective  Benchmarks 

Effective/ 

Non  Effective 

Procedures 

Plebe  Development 
and  Performance 

1.  Academic  Grades 

1.  Y 

1.  Y 

1.  E 

Continues  during 

2.  Military  Performance 
Grades 

2.  Y 

2.  Y 

2.  E 

Academic  Year 

3.  Y 

3.  N 

3.  E 

3.  Attrition  Rate  during 
Academic  Year 

4.  Y 

4.  N 

4.  E 

5.  Y 

5.  Y 

5.  E 

4.  Honor/Conduct 

5.  PRT  Scores 

d.  Benchmark  Analysis 

The  indicators  for  this  objective  have  strong  benchmarks.  They  are  well 
defined  and  are  easy  to  collect  and  apply  class- wide.  Moreover,  the  benchmarks  provide 
for  easy  comparison  with  previous  classes  or  pre-determined  targets.  As  reported  earlier, 
conduct  and  honor  offenses  have  no  minimum  targets  by  can  be  easily  applied  to 
comparisons  with  previous  classes. 

Grade  point  average  and  Military  performance  grades  (MQPR)  are 
reported  and  recorded  in  the  MIDS  database.  A  minimum  GPA  and  MQPR  of  2.0  out  of 
4.0  is  required  to  pass.  Minimum  scores  for  the  PRT  vary  with  age  and  gender. 
However,  specific  minimum  scores  are  identified.  Moreover,  PRT  scores  can  be 
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collected  class- wide  for  comparison  against  pre-selected  targets  or  previous  plebe 
performance. 

Attrition  rates  and  incidence  of  honor/conduct  offences  are  collected  in  the 
MIDS  database.  There  are  no  minimum  benchmarks  for  these  indicators.  The  data 
collected  is,  however,  easily  compared  with  previous  plebe  classes. 

e.  Conclusions 

There  are  no  specific  performance  measurement  procedures  defined  for 
this  objective.  Nevertheless,  the  ad  hoc  performance  measurement  methods  applied  to 
this  objective  by  academy  personnel  are  more  successful  in  assessing  objective  success 
than  the  previous  three  objectives. 

The  objective  and  its  outcome  are  very  well  articulated  and  contain  no 
undefined  terms.  This  allows  academy  personnel  to  better  identify  indicators,  data 
sources,  and  benchmarks. 

Data  is  collected  that  is  easily  accessed  and  provides  for  easy  comparison 
inter-class  as  well  as  intra-class.  Furthermore,  clear  performance  benchmarks  are 
available  for  three  of  five  indicators  that  provide  easy  comparison  with  previous 
performance  or  pre-selected  targets. 

D.  CHAPTER  SUMMARY 

This  chapter  analyzed  the  performance  measurement  and  outcome  management 
procedures  used  to  assess  the  success  of  the  Plebe  Summer  program  in  meeting  its 
objectives.  Four  objectives  were  identified  followed  by  an  analysis  of  the  performance 
measurement  and  outcome  management  procedures  applied  to  each  objective. 

Only  one  of  four  objectives  is  supported  by  effective  performance  measurement 
and  outcome  management  procedures.  Moreover,  each  objective  is  supported  by 
procedures  that  are  improvised  due  to  the  lack  of  defined  performance  measurement  and 
outcome  measurement  procedures  in  Plebe  Summer  governing  instructions. 

The  following  chapter  provides  a  summary  of  the  complete  research,  research 
conclusions,  and  recommendations  for  the  Naval  Academy  and  further  research. 
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VI.  SUMMARY,  CONCLUSIONS,  AND  REOMENDATIONS 


A.  SUMMARY 

The  purpose  of  this  research  was  to  assess  the  effectiveness  of  the  Plebe  Summer 
performance  measurement  and  outcome  management  procedures  in  measuring  the 
success  of  Plebe  Summer  objectives.  First,  the  United  States  Naval  Academy  and  its 
mission  were  introduced.  The  Academy  has  been  the  premier  source  of  Naval  officers 
for  over  150  years  focusing  on  the  devebpment  of  young  men  and  women  morally, 
mentally,  and  physically.  The  role  of  Plebe  Summer  as  the  foundation  of  midshipman 
development  was  then  presented.  Plebe  Summer  is  a  diverse  program  of  cultural 
indoctrination  and  training  designed  to  produce  fourth-class  midshipmen  who  function 
effectively  in  the  Academy’s  brigade  of  midshipmen.  Plebe  Summer  is  the  first  military 
training  evolution  for  most  members  of  the  incoming  plebe  class  (USNA,  2001b,  p.  1-1). 

The  principles  and  concepts  of  performance  measurement  and  outcome 
management  were  next  introduced  as  a  management  tool  designed  to  assist  program 
managers  dealing  with  increasingly  complex  systems.  Performance  measurement,  in 
short,  is  the  process  of  assessing  progress  toward  organizational  goals,  including  the 
efficiency  with  which  resources  are  changed  into  outputs,  the  quality  of  outputs, 
outcomes  achieved  and  the  overall  effectiveness  of  the  organization’s  efforts  toward  their 
mission  (NPR,  1997,  p.  4). 

Research  shows  that  the  concepts  of  performance  measurement  can  assist 
personnel  leading  programs  such  as  Plebe  Summer.  Zigon  (1998)  reports  four  basic 
motives  to  incorporate  performance  measurement  as  a  part  of  any  management  system:  1) 
you  cannot  manage  what  you  cannot  measure,  2)  you  cannot  improve  what  you  cannot 
measure,  3)  high  performance  teams  require  clear  goals,  and  4)  reward  systems  require 
metrics.  Additional  research,  directed  specifically  toward  public  sector  management, 
indicates  that  management  based  on  performance  measurement  systems  provide  greater 
accountability,  increased  service  to  the  public,  and  increased  public  participation  (NCPP, 
1997,  p.  3). 
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The  research  began  with  a  thorough  literature  review  of  all  Naval  Academy 
instructions  governing  Plebe  Summer  and  current  literature  encompassing  performance 
measurement  and  outcome  management.  The  literature  review  had  two  purposes.  The 
first  purpose  was  to  identify  Plebe  Summer  objectives  and  performance  measurement 
techniques  defined  by  Naval  Academy  instruction.  The  second  purpose  of  the  literature 
review  was  to  examine  the  latest  research  and  practical  applications  of  performance 
measurement  in  public  and  private  sectors. 

Next  pre- interview  questionnaires  and  semi- structured  interviews  of  six  Plebe 
Summer  experts  were  conducted.  The  pre- interview  questionnaires  and  interviews  were 
designed  using  Hatry  and  Kopczynski’s  (1997)  Guide  to  Program  Outcome 
Measurement.  The  purpose  of  the  questionnaires  and  interviews  was  to  collect  data  from 
Plebe  Summer  experts  concerning  the  identification  and  use  of  performance  measurement 
and  outcome  management  procedures  as  they  are  practically  applied  by  to  measuring  the 
success  of  attaining  Plebe  Summer  objectives.  The  literature  review  and  interviews 
defined  four  Plebe  Summer  objectives: 

•  Effect  the  transition  from  civilian  or  enlisted  to  officer  candidate  status. 

•  Indoctrinate  and  train  midshipmen  in  proper  military  discipline,  courtesy, 
requirements  and  traits  of  special  trust  and  confidence,  teamwork,  and 
personal  accountability. 

•  Provide  basic  training  in  seamanship,  marksmanship,  military  drill,  watch 
standing,  and  customs  and  traditions  of  the  Naval  service  (USNA,  2001b,  p. 
6). 

•  Prepare  the  Plebe  class  for  integration/incorporation  into  the  Brigade  of 
Midshipmen  and  the  challenge  of  Plebe  academic  year. 

Naval  Academy  performance  measurement  and  outcome  management  techniques 
applied  to  each  objective  were  then  evaluated  using  criteria  generated  by  Hatry  and 
Kopczynski’s  (1997)  Guide  to  Program  Outcome  Measurement  for  the  U.S.  Department 
of  Education.  Experts  provided  information  concerning  the  outcomes,  outcome 
indicators,  data  collection  methods,  and  benchmarks  applied  to  each  objective. 
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B.  CONCLUSIONS 


The  Plebe  Summer  performance  measurement  procedures  are  inadequate.  There 
is  no  specific  performance  measurement  program  designed  to  support  Plebe  Summer 
evaluation.  The  terms  and  definitions  associated  with  Plebe  Summer  objectives  are  not 
clear  to  all  program  staff.  Moreover,  only  three  of  four  objectives  are  defined  by 
governing  Plebe  Summer  instructions.  Although  data  collection  and  evaluation  occurs,  it 
is  based  on  individual  performance  rather  than  program- wide  evaluation.  Additionally,  no 
quality  benchmarks  are  defined  that  supports  a  qualitative  evaluation  of  success. 
Therefore,  ad  hoc  procedures  implemented  by  Plebe  Summer  experts  are  insufficient  to 
assess  successful,  class- wide,  objective  achievement. 

Terms  such  as  “officer  candidate  status”,  “special  trust  and  confidence”,  and 
“teamwork”  must  be  better  defined.  Lacking  definitive  terms,  Naval  Academy  staff  is 
unable  to  identify  specific  outcomes  that  should  be  monitored  as  called  for  by  Hatry  and 
Kopczynski’s  (1997). 

The  absence  of  the  fourth  objective,  (preparing  the  midshipmen  for  integration 
into  the  brigade  of  midshipmen)  from  Plebe  Summer  instructions  places  a  great  deal  of 
stress  on  Plebe  Summer  leadership  in  measuring  the  objective’s  successful  achievement. 
Even  though  the  last  objective  was  not  defined  by  Plebe  Summer  instructions,  it  is  the 
only  one  of  the  four  identified  objectives  that  is  supported  by  effective  performance 
measurement  and  outcome  management  procedures.  Therefore,  it  is  the  only  objective 
with  which  Plebe  Summer  leaders  are  able  to  make  a  qualitative  judgment  on  whether  the 
objective  was  successfully  met  or  not.  Perrin  (1998)  described  how  varying 
interpretations  of  terms  and  concepts  result  in  measures  becoming  meaningless  in  the 
long  term.  This  is  an  issue  for  this  Plebe  Summer  objective  because  it  is  an  improvised 
objective  and  not  defined  by  instruction.  Outcomes,  outcome  indicators,  data  collection, 
and  benchmarks  used  for  performance  measurement  of  this  objective  are  applied 
unevenly  throughout  the  brigade  and  in  an  ad  hoc  fashion. 

Data  is  currently  collected  for  a  variety  of  plebe  activities  such  as  PRT,  academic 
tests,  military  performance,  rate  knowledge,  and  marksmanship.  However,  in  most  cases, 
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the  data  collected  is  not  compiled  in  a  central  database  to  represent  class-wide 
performance.  This  results  in  the  inability  to  assess  program  effectiveness.  Lacking  class¬ 
wide- data,  outcome  indicators  are  irrelevant  because  they  do  not  support  the  program- 
wide-objectives  specified  in  governing  instructions. 

C.  RECOMMENDATIONS 

1.  For  the  Naval  Academy 

Plebe  Summer  is  an  important  component  in  achieving  the  Naval  Academy’s 
mission.  This  study  suggests  that  the  Naval  Academy  can  apply  performance 
measurement  and  outcome  management  to  assess  the  quality  of  Plebe  Summer.  The 
Naval  Academy  should  establish  a  working  group  to  oversee  the  development  of  an 
outcome  measurement  process  for  the  Plebe  Summer  program.  The  following 
recommendations  outline  how  the  Naval  Academy  can  assist  a  working  group  in  this 
endeavor. 

The  first  step  in  developing  an  effective  use  of  Plebe  Summer  data  is  to  better 
define  Plebe  Summer  objectives.  While  three  of  four  objectives  are  described  through 
the  governing  instructions,  the  purposes  of  each  objective  are  not  defined.  Moreover, 
undefined  terms  require  ad  hoc  interpretation  by  staff.  The  objectives  should  be  modified 
in  order  to  clarify  expectations.  This  will  allow  a  working  group  to  more  accurately 
identify  the  outcomes,  outcome  indicators,  and  benchmarks  required  to  support  an 
effective  measurement  process. 

Data  collection  procedures  that  support  programwide  assessment  should  be 
designed.  Benchmarks  are  rendered  ineffectual  without  adequate  class-wide  data  that 
supports  comparison  among  and  between  years.  An  important  aspect  of  Plebe  Summer 
data  is  not  collected.  Specifically,  no  data  is  collected  on  the  opinions  of  plebes  as  the 
recipients  of  Plebe  Summer  training.  The  Naval  Academy  should  develop  a  method  to 
survey  plebes  on  their  attitudes  and  opinions  concerning  the  effects  of  the  curriculum  in 
preparing  them  for  the  challenges  of  Plebe  Year.  Topics  such  as  teamwork,  special  trust 
and  confidence,  and  discipline  can  be  assessed  using  such  a  survey  method. 
Additionally,  the  academy  should  survey  staff  and  upper-class  midshipmen  in  order  to 
assess  their  opinions  of  program  wide  performance  and  the  performance  of  the  plebe 
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class  during  the  academic  year.  For  instance,  staff  and  upper-class  midshipmen  can 
provide  valuable  data  on  what  areas  of  plebe  performance  require  further  development. 
This  type  of  data  would  assist  a  working  group  in  developing  and  recommending  Plebe 
Summer  program  improvements. 

Benchmarks  must  be  identified  by  Naval  Academy  leadership.  No  performance 
assessment  can  occur  in  the  absence  of  useful  comparison  structures.  While  PRT  scores 
and  Grade  Point  Averages  have  minimum  allowable  scores,  the  academy  has  not 
specified  any  level  of  quality  performance  that  supports  comparison  between  plebe  years. 
The  question  “is  the  plebe  class  performance  good  or  bad?”  cannot  be  answered. 
Furthermore,  benchmarks  identifying  pre-selected  targets  are  not  defined.  A  working 
group  could  recommend  possible  benchmarks;  however  the  academy  leadership  must 
define  specific  levels  of  performance  qualifications. 

2.  Recommendations  for  Further  Research 

This  research  focused  on  the  performance  measurement  and  outcome 
measurement  procedures  applied  to  Plebe  Summer.  There  are  additional  research  areas 
that  emerged  throughout  the  conduct  of  this  study.  The  following  paragraphs  describe 
some  additional  recommended  research.  There  is  no  detailed  performance  measurement 
program  specifically  designed  to  support  Plebe  Summer.  Most  performance 
measurement  and  outcome  management  is  conducted  using  improvised  procedures. 
Therefore,  the  additional  recommended  research  can  be  conducted  with  the  precept  of 
developing  such  a  performance  measurement  system. 

No  study  has  been  conducted  that  measures  the  opinions  of  plebes  concerning 
their  feelings  toward  the  effectiveness  of  Plebe  Summer  in  preparing  them  for  the 
challenges  of  the  Naval  Academy.  This  study  recommends  the  development  of  a  survey 
to  identify  the  opinions  and  attitudes  of  plebes  towards  the  Plebe  Summer  curriculum. 
Plebe  attitudes  towards  the  Plebe  Summer  curriculum  should  be  studied  before  such  a 
survey  is  developed.  The  purpose  of  the  study  would  be  to  answer  the  questions,  “What 
areas  of  the  Plebe  Summer  curriculum  do  plebes  identify  as  valuable?”  and  “Do  plebes, 
upper-class  midshipmen,  and  academy  staff  share  common  opinions  of  Plebe  Summer 
curriculum  value?” 
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The  absence  of  benchmarks  was  a  common  theme  throughout  the  research.  A 
study  could  be  conducted  that  would  identify  levels  of  performance  as  they  pertain  to 
Plebe  Summer.  A  study  such  as  this  would  identify  benchmarks  and  performance 
qualifiers  such  as  “excellent”  or  “outstanding”  that  can  be  attached  to  class-wide 
performance.  This  study  could  answer  the  question,  “How  can  plebe  performance  be 
classified?” 
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APPENDIX  A  -  PLEBE  SUMMER  TRAINING  TOPICS 


I.  Command  Religious  Program 


Category 

Desig 

Brief  Description 

n 

Brief 

CRP  1 

Command  Religious  Program,  Plebe  Summer 
Indoctrination 

1 

Inst 

IRT 

Interpersonal  Relations  Training  (Three 
sessions) 

3 

OpEvnt 

- 

Worship  Services  (Optional) 

N/A 

OpEvnt 

- 

Plebe  Hour  (Optional) 

N/A 

Brief 

CRP  2 

Command  Religious  Programs  Academic  Year 
Indoctrination 

1 

Brief 

Holocaust 

Holocaust  Museum  Brief 

1 

Total 

6 

II.  Physical  Education 


Category 

Desig 

Brief  Description 

Time 

Req’d 

Brief 

PEP  INDOC 

Introduction  to  PEP 

1.5 

Inst 

PEP 

Physical  Education  Program 

36 

Exam 

IPRT 

Initial  Physical  Readiness  Test 

1.5 

Exam 

IPRT 

Intermediate  Physical  Readiness  Test 

2 

Exam 

FPRT 

Final  Physical  Readiness  Test 

2 

Inst 

USMC  O/C 

Marine  Corps  Obstacle  Course 

2 

Exam 

USMC  O/C 

Marine  Corps  Obstacle  Course 

Timed 

2 

Inst 

see 

Squad  Combat  Course 

2 

Inst 

SER  (2) 

Squad  Endurance  Run 

4 

Inst 

Crew/Box/ 

Gym/Wres 

Introduction  to  Competitive 

Sports  and  Close  Quarters  Combat 

5 

Inst 

HTH 

Hand  to  Hand  Combat  Instruction  (1-3) 

3 

Inst 

PCON 

Personal  Conditioning  (1-3) 

3 

Inst 

SWIM 

Swimming  Instruction  (1-7) 

7 

Inst 

TAC 

Tarzan  Assault  Course 

3 

Inst 

CLW 

Climbing  Wall  (3) 

6 

Brief 

- 

Intramural/Intercollegiate  Sports 

2.5 

Brief 

PEREQ 

Physical  Education  Requirements 

1 
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OpEvnt  Sports  Period  Intramural  Competitions  42 


OpEvnt  -  Basketball  Competition  2 


OpEvnt  -  Soccer  Competition  2 


OpEvnt  -  Wrestling  Competition  2 


OpEvnt  -  Boxing  Smoker  2 


OpEvnt  All  Comers  Track  Meet  3.5 


Total  137 


III.  Academic 


Category 

Desig 

Brief  Description 

Time 

Reqd 

Exam 

- 

Placement  Validation  Exams 

Var 

Exam 

LASSI 

Learning  and  Study  Strategies  Inventory 

1 

Brief 

- 

Associate  Dean  Academic  Affairs 

1 

Brief 

- 

Academic  Dean  and  Provost  Remarks 

1 

Inst 

ACR 

Advising  Period 

2 

Inst 

- 

Ac  Handbook 

1 

Inst 

- 

Ac  Advisor/Study  skills 

2 

AdmnEvnt 

- 

Computer  Orientation 

2 

AdmnEvnt 

- 

ACE/MB  TI 

2.5 

Total 

12.5 

IV.  Supply 


Category 

Desig 

Brief  Description 

Time 

Reqd 

AdmnEvnt 

HCT 

Haircuts  (1-3) 

3 

AdmnEvnt 

BI 

Book  Issue 

1 

AdmnEvnt 

RTSA  1 

Repair  Tailor  Shop  Alteration  I 

1 

AdmnEvnt 

RTSA2 

Repair  Tailor  Shop  Alteration  II 

1 

AdmnEvnt 

RTSA  3 

Repair  Tailor  Shop  Alteration  III 

1 

AdmnEvnt 

DUIC 

Contract  Tailor  Shop 

1 

AdmnEvnt 

USI 

USNA  Store  Issue 

2.75 

AdmnEvnt 

USX 

USNA  Store  Exchange 

1 

AdmnEvnt 

- 

Uniform  Issue  Verification 

1 

AdmnEvnt 

- 

Computer  Issue 

3 

Total 

15.75 
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V.  Administrative 


Category 

Desig 

Brief  Description 

AdmnEvnt 

PHOTO 

Photographs  for  Performance/ Academic  Records 

1 

AdmnEvnt 

- 

Urinalysis 

2.5 

AdmnEvnt 

ID 

Military  ID  Processing/Finger  Printing/EPSQ 
Review 

2 

AdmnEvnt 

EPSQ 

EPSQ  Brief 

1 

Total 

6.5 

VI.  Medical/Dental 


Category 

Desig 

Brief  Description 

■m 

Exam 

DEN  1 

Dental  Exam  1 

2 

Exam 

DEN  2 

Dental  Exam  2 

1 

Exam 

EYES 

Eye/Hearing  Exam 

3 

Inst 

FIRST  AID 

First  Aid  Training 

2 

Brief 

- 

Health,  Hygiene  and  STD  Brief 

1 

Brief 

- 

TRICARE  Brief 

1 

Brief 

- 

Smoking  Cessation  Brief 

1 

Brief 

- 

Drug  and  Alcohol  Brief 

1 

Brief 

- 

Eating  Disorders  Brief 

1 

Total 

13 

VII.  Professional  Development 


Category 

Desig 

Brief  Description 

Brief 

ProDev 

Department  of  Professional  Development 
Introduction  Brief 

1 

Brief 

CIP 

Career  Brief 

2 

Inst 

DC 

Damage  Control  1-3  (P-100) 

6 

Inst 

SIG 

Signaling  1-4  (P-100) 

7 

Inst 

KA 

Sailing  Knockabouts  1-4  (P-100) 

11 

Inst 

LA 

Sailing  Lasers  1-3  (P-100) 

9 

Inst 

NA44 

Sailing  Navy  44’s  (P-100) 

6 

Inst 

Study 

Study  Time 

3 

Total 

45 
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VIII.  Commandant 


Category 

Desig 

Brief  Description 

Brief 

- 

Superintendent’s  Call 

1 

Brief 

- 

Commandant’s  Calls 

2 

Brief 

- 

OIC  Calls 

2 

Brief 

- 

Character  Development  Officer’s  Call 

1 

Brief 

- 

Regimental  Commander’s  Call 

2 

Brief 

- 

Distinguished  Speaker  Series 

3 

Brief 

SAVI 

SAVI  BRIEF 

1 

Brief 

SAVI 

SAVI  TRAINING  (2) 

2 

Brief/ 

Inst 

SELT 

Senior  Enlisted  Time 

1 

Brief 

CCT 

Company  Commander  Time 

VAR 

AdmnEvnt 

- 

Personal  Time 

VAR 

AdmnEvnt 

- 

Counseling  Time 

VAR 

AdmnEvnt 

RATE 

Rate  Review  Period 

VAR 

AdmnEvnt 

- 

Blue  and  Gold 

VAR 

Inst 

HONOR 

Character  Education/ 

Honor  Lessons  (1-7) 

9 

Inst 

CD 

Character  Education/ 

Character  Lessons  (1-6) 

6 

Inst 

NL102 

Naval  Leadership  102 

9 

Brief/ 

Inst 

HISTORY 

Military  History  and  Traditions 

4 

AdmnEvnt 

RIFLE 

Rifle  Issue/Turn  In 

1 

Inst 

COD 

Close  Order  Drill 

28 

Inst 

- 

Practice  Parades 

10 

OpEvnt 

- 

Formal  Parades 

4 

OpEvnt 

- 

Platoon  Drill  Evaluation 

1 

OpEvnt 

- 

Platoon  Drill  Competition 

1 

Inst 

WTP 

Weapons  Training 

11 

Brief 

Sponsor 

Sponsor  Program  Brief 

1 

Brief 

D&B 

Drum  and  Bugle  Corps  Auditions 

1 

Brief 

MA 

Music  Activities  Auditions 

1 

Brief 

- 

Alumni  Association  Introduction 

1 

Brief 

MUSEUM 

USNI  Museum  Orientation 

1 

Brief 

LIBRARY 

Library  Orientation 

1 

Inst 

CODE 

Code  of  Conduct  Instruction  (1-2) 

2 

Inst 

ETIQUETTE 

Service  Etiquette  Instruction 

1 

Brief 

JPJ 

John  Paul  Jones  Crypt  Orientation 

1 
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Inst 

SLI 

Squad  Leader  Instruction  Periods  (1-22) 

22 

Inst 

SLT 

Squad  Leader  Training  Periods  (1-5) 

5 

Inst 

RP 

Room  Inspection  Preparation 

5 

Exam 

- 

Formal  Room  Inspection 

5 

Inst 

UP 

Personnel  Inspection  Preparation 

3 

Exam 

- 

Formal  Personnel  Inspection 

3 

Exam 

- 

Sea  Bag  Inspection 

1 

AdmnEvnt 

PHONE 

Phone  Calls 

3 

AdmnEvnt 

360  FB 

360  degree  feedback 

2 

OpEvnt 

- 

Fourth  of  July  Program 

2 

OpEvnt 

- 

8th  &  I  Sunset  Parade 

2 

OpEvnt 

- 

Plebe  Rate  Competition/Sing  Along 

2 

OpEvnt 

- 

Detail  Set  Turnover 

1 

OpEvnt 

- 

Regimental  Picnic/Field  Meet 

5 

OpEvnt 

- 

Holocaust  Museum/Vietnam  Memorial/Navy 
Memorial  Visit 

10 

OpEvnt 

- 

Baseball  Game 

7 

OpEvnt 

- 

Formal  Regimental  Dinner 

2 

OpEvnt 

- 

Parents’  Weekend 

N/A 

Total 

189 
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APPENDIX  B  -  SQUAD  LEADER  INSTRUCTION  LESSONS 


Lesson  Title 


Short  Description 


1.  Plebe  Rates/Oath  of  Office 

2.  Watch  standing 

3.  Personal  Hygiene 

4.  Bancroft  Hall  Orientation 

5.  Yard  Tour 

6.  Honor  I 

7.  Naval  Customs  and  Traditions 

8.  Naval  Organization 

9.  Mission  of  the  Naval  Academy 

10.  COC/Followership 

11.  Loyalty/Communication 

12.  Good  Order  and  Discipline 

13.  Leadership  Development 

14.  Enlisted  Personnel  at  USNA 

15.  Military  Performance  System 

16.  Admin  Conduct  System 

17.  Honor  II 


An  overview  of  plebe  rates  along  with  a  detailed 
discussion  of  the  Oath  of  Office. 

Presentation  of  Watch  standing  procedures. 

Basic  practices  of  personal  hygiene  and  proper  care 
and  prevention  of  foot  blisters  and  skin  rashes. 

Discussion  and  tour  of  the  Bancroft  Hall  layout. 

A  walking  tour  of  key  buildings  and  monuments  in 
the  Yard 

Introduction  to  the  Naval  Academy  Honor  Concept. 

An  overview  of  Naval  and  Marine  Corps 

history,  heritage,  and  traditions. 

An  overview  of  the  organizational  structure  of  the 
Department  of  the  Navy. 

Review  and  gain  a  more  in  depth  understanding  of 
the  mission  of  the  Naval  Academy. 

Review  the  structure,  purpose,  and  benefits  of  the 
chain  of  command  and  good  followership. 

Review  concepts  of  initiative,  loyalty,  and 
communication. 

A  discussion  of  the  importance  of  good  order  and 
discipline  to  a  unit. 

A  discussion  of  leadership  and  how  professional 
development  is  accomplished  at  USNA 

Review  of  the  role  of  enlisted  personnel  at  USNA 

A  discussion  of  the  Military  Performance  System, 
MQPR,  and  Fitreps. 

A  discussion  of  the  Administrative  Conduct 
System,  rules,  and  regulations. 

A  discussion  of  the  ethical  standards  required  of 
USNA  Midshipmen 
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Lesson  Title 

Short  Description 

18. 

Military  Etiquette 

Explanation  of  the  basic  principles  of  military 
etiquette  and  protocol. 

19. 

Main  Office  Watch  standing 

A  discussion  of  the  basic  requirements  to  standing  a 
Main  Office  watch. 

20. 

Leave/Liberty 

A  review  of  Midshipmen  leave  and  liberty  policies, 
and  Parent’s  Weekend  regulations. 

21. 

Study  Skills/Academics 

A  discussion  of  the  importance  of  academics  and 
study  method  suggestions. 

22. 

Joining  the  Brigade 

Acquaint  Plebes  with  knowledge  necessary  to 
transition  to  the  Academic  Year. 
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APPENDIX  C  -  EXPERT  PRE-INTERVIEW  WORKSHEET  AND 

DEFFINITIONS 

Expert  Pre -Interview  Worksheet 

The  purpose  of  this  interview  is  to  develop  an  understanding  of  how  Plebe 
Summer  success  is  measured.  This  interview  is  being  done  as  part  of  an  analysis  of 
performance  measures  used  in  Plebe  Summer  for  my  Master’s  Thesis.  This  interview 
will  attempt  to  identify  the  following: 

a.  The  objectives  of  Plebe  Summer 

b.  The  outcomes  each  objective  is  designed  to  achieve 

c.  Indicators  of  outcome  success 

d.  Data  sources  for  outcome  indicators 

e.  Performance  Benchmarks  for  each  outcome  indicator 

Although  Plebe  Summer  has  dual  purposes,  my  thesis  concentrates  only  on  the 
training  of  incoming  plebes.  The  leadership  training  of  upper  class  midshipmen  is  not 
included  in  this  study. 

This  interview  is  not  an  opinion  poll.  I  want  to  identify  exactly  how  the  Naval 
Academy  decides  whether  Plebe  Summer  is  successful  or  not  (and  at  what  level).  This 
pre- interview  is  designed  to  provoke  thought,  not  limit  scope.  If  you  feel  there  are  items 
(objectives,  outcomes,  indicators,  etc)  missing  or  requiring  expansion  please  add  them. 

Key  questions  to  ask  yourself  as  you  work  through  the  worksheet  are:  What 
information  is  important  to  me  in  judging  Plebe  Summer  success?  What  are  the  ultimate 
objectives  of  Plebe  Summer?  How  do  I  know  if  an  objective  is  being  met? 

The  matrix  attached  to  this  coversheet  lists  the  objectives  of  Plebe  Summer  as 
stated  in  COMDTMIDNINST  1510.2B.  Following  each  objective  is  a  list  of  outcomes, 
outcome  indicators,  data  sources  and  benchmarks.  In  many  instances  no  information  is 
listed.  I  ask  for  your  expert  opinion  on  filling  in  the  missing  data. 
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Each  objective  should  have  one  or  more  associated  outcomes.  In  turn,  each 
outcome  should  be  aligned  with  one  or  more  outcome  indicators,  data  sources,  and 
benchmarks.  The  definitions  of  these  terms  are  attached. 

As  you  work  through  the  matrix  please  consider: 

a.  In  your  opinion,  does  this  list  of  objectives  cover  all  the  objectives  of 
Plebe  Summer? 

b.  In  your  opinion,  what  are  the  outcomes  these  objectives  intended  to 
achieve? 

c.  What  are  the  indicators  of  each  outcome? 

d.  How  is  data  on  each  indicator  collected? 

e.  What  is  the  benchmark  for  each  indicator?  What  are  the  minimum 
acceptable  levels  of  performance  associated  with  each  indicator? 
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DEFINITIONS 


Outcome:  Outcomes  are  not  what  the  program  itself  did.  Rather,  outcomes  are  the 
consequences  of  what  the  program  did.  They  provide  information  on  events, 
occurrences,  conditions,  or  changes  in  attitudes  and  behavior  that  indicate  progress 
toward  achievement  of  the  mission  and  objectives  of  the  program  Outcomes  happen 
outside  the  program. 

There  are  two  types  of  outcomes  Intermediate  Outcomes  and  End  Outcomes. 

Intermediate  outcomes  are  expected  to  lead  to  an  ends  desired,  but  are  ends  in 
themselves.  Examples  are  participation  in  program,  improvements  in  learning  practices 
relating  to  instruction,  and  increase  in  hours  of  study. 

End  outcomes  are  the  desired  results  of  the  program.  These  can  be  either  short-term  or 
long-term  Increased  test  scores,  reduction  of  conduct  offenses,  reduced  attrition  rates. 


Objectives:  This  is  often  attached  to  the  word  Mission.  It  is  a  statement  that  tells  where 
the  program  wants  to  go.  What  are  the  overarching  results  that  are  planned? 

Outcome  Indicators:  This  is  not  the  same  as  Outcomes.  Outcome  indicators  specify 
what  is  to  be  measured.  Specific  indicators  will  depend  on  the  type  of  data  collection 
used.  Outcome  indicators  identify  specific  numerical  values  that  indicate  progress 
toward  achieving  an  outcome  (such  as  percentage  or  ratio).  Examples  include  percentage 
of  persons  qualifying  expert  rifleman. 

Benchmark:  What  do  you  compare  outcome  indicator  information  to?  Benchmarks 
provide  a  comparison  for  data.  Benchmarks  provide  the  minimum  level  of  performance 
or  change  acceptable.  Major  types  of  benchmarks  include: 

Previous  Performance  (Improvement  over  time) 
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Performance  of  Similar  Group  (Comparison  against  the  best) 
Outcomes  of  different  groups  (comparison  against  the  best) 
Pre-selected  targets  (pre-selected  standard) 

Different  delivery  practices  (use  of  comparison  groups) 
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